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of Education Sciences of the U.S. Department of Education. The Commissioner of Education Statistics is 
responsible, by law, for carrying out the NAEP project through competitive awards to qualified organizations. 
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The National Assessment of Educational Progress (NAEP) 
is tire nation’s ongoing representative sample survey of 
student achievement in core subject areas. NAEP, Imown as 
tlie Nation’s Report Card, is autliorized by Congress and 
administered by the National Center for Education Statistics 
(NCES) of the Institute of Education Sciences in the LIS. 

Department of Education. NAEP regularly reports to tlie 
public on the educational progress of students in grades 4, 8, 
and 12. 

In 2002, NAEP assessed tlie reading and writing 
performance of the nation’s fourth-, eighth-, and twelfth- 
grade students. N7\Ed^ also conducted assessments of 
fourtli- and eighth-graders’ reading and writing in most of 
the states. 

In 2001, after discussion among NCES, the National 
Assessment Governing Board (NAGB), and the leadership 
of the Council of the Great City Schools, Congress 
appropriated funds for a trial district-level assessment and 
NAGB passed a resolution approving tlie selection of five 
large urban districts for participation in the Trial Urban 
District Assessment, a special project within NAET. This 
report presents results of NAEP’s Trial Urban District 
Assessment in writing for public school students in the 
following participating urban school districts: Adanta City, 

Cliicago School District 299, Houston Independent School 
District, Los Angeles Lfnified, New York City Public Schools, 
and Washington, DC. This represents NAEP’s first 
assessment of urban districts based on samples specially 
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designed to allow reporting of subgroup 
data. The first five districts participated 
voluntarily in the N^\EP 2002 writing 
assessment at grades 4 and 8. Results tor the 
District of Columbia, which in this and past 
N/VEP assessments has been sampled and 
assessed along with states and other jurisdic- 
tions, are also included in this report. Data 
for public schools across the nation and for 
central city public schools 

arc provided for comparison purposes.^ 

The public schools sampled also included 
charter schools, which in some cases were 
not managed by the urban school districts. 

N^\EP does not provide scores for 
individual students or schools. It reports 
results for groups of students (e.g., tourth- 
graders). For each group on each table in 
the report, assessment results are described 
in one of two ways. First, the group’s 
average writing score is reported on a scale 
from 0 to 300. Performance for each grade 
is scaled separately; therefore, average scale 
scores cannot be compared across grades. 
The term ''average score” is used through- 
out this report to refer to the average scale 
score on the N^\EP writing scale. Second, 
student writing performance is reported in 
terms of tlie percentage of students in die 
group who reached each of three achieve- 
ment levels: Basic) Brofident, m'ld A^dvanced. 
The Proficient \evt\ for each grade is defined 
by NAGB as representing "solid academic 
performance,” which demonstrates "compe- 
tency over challenging subject matter” for 
the grade assessed. Basic indicates partial 
mastery of skills that are fundamental for 
proficient work. Advanced denotes superior 
performance. 



The achievement levels ;ire performance 
standards adopted by the National Assess- 
ment Governing Board (NAGB) as part of 
its statutory responsibilities. The achieve- 
ment levels are a collective judgment of 
what students should know and be able to 
do for each grade tested. As provided by 
law, NCES, upon review of a congression- 
ally mandated evaluation of N^AEP, deter- 
mined that the achievement levels are to be 
used on a trid basis and should be inter- 
preted with caution. However, both NCES 
and the Board believe that die performance 
standards are useful for understanding 
trends in student achievement. They have 
been widely used by national and state 
officials ;md otiiers as a common yardstick 
of academic performance. 

The results are based on representative 
samples of students for the nation, for 
participating districts, and for students in 
schools in central cities. A central city is a 
city of 50,000 or more that is the largest in 
its metropolitan area, or can otherwise be 
regarded as "central.” The term means "a 
city that is central,” not "the centoil part of 
a city” or the "inner city^” Note that central 
cities encompass wider areas than what is 
commonly referred to as "the inner city.” 
(See further details in appendix C, dype of 
Location). 

In order to obtain reliable data, sufficient 
numbers of the selected schools and stu- 
dents must participate in the assessment. All 
six districts met the NCES participation 
criteria for NAEP samples at grade 4, but 
results for New York City^ schools at grade 8 
are not reported because they did not meet 
the participation criteria. 



^ ‘‘CeatraJ city' is (iellued in cliaptet 2 mid more completely in tlie “Tyjie of Ixication" section of appendix C. 
Central city includes nationally rq^resentative public schmls located in central cities \vitliin metropolitan statistical 
areas as detiued by the Federjd Oilice of Managetneut and Budget. It is not synonymous with “inner city.'* 
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Some students vire identified by the 
school districts as students with disabilities 
and/or limited English proficient students. 
Some of these students are excluded from 
the iissessment, and others are tested with 
accommodations related to their status. 
Three of the six districts identified between 
30 and 52 percent of their students as either 
students with disabilities or limited English 
proficient students. Because the percentage 
of students identified, excluded, and as- 
sessed with accommodations varies across 
the districts, that variability should be tiiken 
into consideration in interpreting the results 
and making comparisons (See appendix A, 
table A.l), 

Throughout this report, differences 
between scores iind between percentages are 
discussed in the text in terms of statistical 
significance. All differences reported are 
significant at the 0.05 level (with appropriate 
adjustments for multiple comparisons). 

Overall Writing Results for 
the Urban Districts 

The following summary first describes 
results in terms of average scale scores and 
then in terms of achievement levels. Aver- 
age results for public schools in the districts 
participating in the Trial Urban District 
Assessment are compared, at grades 4 and 
8, witli public schools in the nation, with 
public schools in central cities, and witli 
each other. 

Average Scale Scores 

Results for Grade 4 Public Schools 

■ The average scores for fourth-graders in 
public schools ranged from 135 in the 
District of Columbia to 153 in New York 
City and the nation. 



■ At grade 4, no statisticdly significant 
differences were detected between the 
average scores for students in Houston 
and New York City and the average score 
for students in the nation, while students 
in Atlanta, Chicago, the District of 
Columbia, and Los ^Vngeles had average 
scores lower than the average score in 
the nation, 

■ At grade 4, the average score for students 
in New York City^ was higher than the 
national average score for students in 
central city schools. The average score for 
fourtli-graders in Houston was not found 
to differ significantly from that for cental 
cities, while the average score for students 
in each of the other districts was lower thaii 
the nationd average score for cental cities. 

■ The average score for students in New 
York City was higher than those in all the 
other participating districts except Hous- 
ton. Ihe average score in Houston was 
higher than the scores in Atlanta, Chi- 
cago, and the District of Columbia, but 
was not found to differ significantly irom 
the average scores in Los Angeles and 
New York City. 

Results for Grade 8 Public Schools 

Results for New York City schools at grade 

8 are not reported because they did not 

meet participation criteria. 

■ The average district scores for eighth- 
graders ranged from 128 in the District 
of Columbia and Los ^Angeles to 138 in 
Houston, 

■ In each of the reported districts, the 
average score of eighth-grade students 
was lower than the average score for 
eighth-grade students in the nation. 
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■ At grade 8, no significant difference was 
detected between the average score for 
students in Houston and the average 
score for students in tlie central city 
schools. The average score in the central 
city schools was higher than the average 
scores in Atlanta, Chicago, the District of 
Columbia, and Los ^\ngeles. 

■ The average score tor students in Pious - 
ton was higher than the average scores in 
Atlanta, the District of Columbia, and 
Los /\ngeles. The average score in Chi- 
cago was not found to differ significantly 
from those in Houston and Atlanta, and 
was higher than the average scores in the 
District of Columbia and Los Angeles. 

Writing Achievement Levels 
Results for Grade 4 Public Schools 

■ At grade 4, the percentages of students 
performing at or above T^roficknt ranged 
from 11 percent in the District of Co- 
lumbia to 27 percent in New York City. 

■ The percentages of fourth-grade students 
performing at or above Proficient in 
Houston ^ind New York City were not 
found to be significantly different from 
the percentages in the nation or in central 
cities. Atlanta, Chicago, the District of 
Columbia, and Los ^\ngeles had lower 
percentages of students at or above 
Prqftdent than the nation and central cities. 

■ At grade 4, the percentage of students 
performing at or above Profitcient in New 
Y'ork City was higher than the percent- 
^tges in four other distnets and not found 
to differ significantly from the percentage 
in Houston. 



■ The percentages of fourth-grade students 
performing at or above Basic ranged from 
73 percent in the District of Columbia to 
85 percent in New York City. In public 
schools across the nation, 85 percent of 
students performed at or above tlie Basic 
level. In central city schools, 81 percent 
performed at or above the Bade level. 

Results for Grade 8 Public Schools 

® At grade 8, the percentages of students 
performing at or above P?''q/iden/ ranged 
from 10 percent in Atlanta and the 
District of Columbia to 19 percent in 
Plouston. Thirty percent of eighth- 
graders in public schools in the nation 
and 22 percent in central city schools 
performed at or above the Proficient level. 

® The percentage of students performing 
at or above Proficient Wc'is higher for the 
nation than for any of the five urban 
districts reported, and higher for central 
cities than for all urban districts except 
Houston, where no significant difference 
was detected. The percentages of eighth- 
graders performing at or above Proficient 
in Chicago and Houston were not found 
to differ significantly from each other, 
and both were higher than the compa- 
rable percentages in the Distnet of 
Columbia ^md Atlanta. 

■ The percentage of eightli-graders per- 
forming at or above ranged from 
64 percent in Los Angeles to 74 percent 
in Houston. Eighty -four percent of 
eighth-graders in public schools in the 
nation and 77 percent in central city 
public schools performed at or above the 
Basic level. 
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Results for 
Student Subgroups 

In addition to providing average scores and 
achievement levels for the nation, for states, 
and, in this report, for districts, N-:\EP 
reports provide results for subgroups of 
students defined by various background and 
contextual chivracteristics (e.g., gender, 
eligibility for free/ reduced-price lunch, and 
level of parents’ education). Performance 
results for subgroups are reported primarily 
as comparisons of district average scores 
with the comparable average scores in 
central cities. 

Gender 

Results for Grade 4 Public Schools 

■ No statistically significant difference was 
detected between the average scores of 
male or female fourth -grade students in 
Houston, New York City and the average 
scores of their counterparts in the central 
city schools. Average scores tor fourth- 
grade male and female students in Atlanta, 
Chicago, the District of Columbia, and 
Los Angeles were lower than the average 
score in central city schools. 

® Female iourth -graders had higher average 
scores than male fourth-graders in each 
of the urban districts. 

Results for Grade 8 Public Schools 

■ The average score for eighth-grade 
female students in Houston was not 
found to be significantly different from 
that of the central city public schools. In 
Atlanta, Chicago, the District of Colum- 
bia, and Los Angeles, the average scores 
for both male and female eighth-graders 
were lower than the average scores for 
their counterparts in central city schools. 

■ In all participating districts, female 
students had higher average writing 
scores than male students. 



Race/Ethnicity 

In each of the urban districts assessed, 

Black or Hispanic students constitute the 
majority or the largest racial/ethnic group. 
This distribution differs from that for the 
national writing assessment, in which White 
students constitute a majority — 60 percent 
of the fourth-grade sample and 64 percent 
of the eighth-grade sample. Black students 
made up more than four- fifths of the 
samples at both grades in Atlanta and the 
District of Columbia and nearly half at 
both grades in Chicago. Hispanic students 
made up about two-thirds of the Los 
Angeles samples at both grades and about 
half of the fourth-graders and more than 
half of the eighth-graders in Houston. In 
New York City, more than two-fifths of the 
fourth-graders were Hispanic and just under 
a third were Black. 

Results for Grade 4 Public Schools 

■ In five of the six urban districts in which 
a reliable comparison could be made, 
WTiite fourth-graders had higher average 
scores than their Black and Hispanic 
counterparts. 

■ Black students in grade 4 in Houston and 
New York City had higher average scores 
than those in the central cities. Black 
fourth-grade students in .Atlanta, Chicago, 
and I.OS Angeles had average scores not 
found to differ significantly from their 
counterparts in central cities. In the 
District of Columbia, Black tourth- 
graders had an average score lower than 
that of their counterparts in central cities, 

■ No significant difference was detected 
between the average score for Hispanic 
fourth -graders in five of the six districts 
and Hispanic fourth -graders’ average 
score in central cities. The average score 
for Hispanic fourth-graders in Los 
Angeles was lower than that in central 
cities taken as a whole. 
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■ Average scores for W^liite fourth-grade 
students in Atlanta, the District of 
Columbia, and New York City were 
higher than the average score for VX^hite 
fourth-grade students in central cities. 

■ Average scores for Asian/Pacific Islander 
students in Los Angeles and New York 
City were not found to be significantly 
different from the average score for their 
counterparts in central cities. 

Results for Grade 8 Public Schools 

■ White eighth -graders had higher average 
scores rh;m Black eighth-graders, in every 
district except the District of Columbia, 
where the sample sizt was insufficient to 
permit a reliable comparison. White 
students at grade 8 also had higher 
average scores than Hispanic students in 
Chicago, Houston, and Los Angeles. 

■ The average score for eighth-grade Black 
students in the District of Columbia was 
lower than that for Black eighth-grade 
snidents in the central city schools, and 
no significant difference was detected 
between the average score for Black 
studente in any of the other four districts 
and the national average score for Black 
students in central cities. 

■ Average scores for Hispanic students 
were not found to differ significantly 
between the districts and the national 
average for central cities, except in Los 
Angeles, where Hispanic students had a 
lower av'^erage score than their counter- 
parts in central cities. 

■ The average score for Wliite eighth -grade 
students in Houston was higher than that 
of XX'Tiite students in the central city 
schools, while the average score in Los 
Angeles was lower. 



Eligibility for 

Free/Reduced'Price Lunch 

The National School Lunch Program 
providing free/reduced-price lunch is 
administered by the U.S. Department of 
Agriculture (LJSDA) for children near or 
below the poverty line. Eligibility is deter- 
mined by the USDAs Income Eligibility 
Guidelines (http:/ /wv\w. fns.usda.gov/cnd/ 
lEGs&NAPs/IEGs.htm). 

Results for Grade 4 Public Schools 

At grade 4, rates of student eligibility for 
free /reduced -price lunch ranged from 70 
percent in New York City to 89 percent in 
Chicago. 

■ Fourth -grade students eligible for free/ 
reduced-price lunch had lower average 
scores than those not eligible in every 
district except Los Angeles and New 
York City, where no significant difference 
between the two eligibility^ categories was 
detected. 

■ Fourth-gmde students eligible for free/ 
reduced-price lunch in New York City 
had a higher average score than tlie 
national average score for their counter- 
parts in central cities, while students in 
Chic:igo and the District of Columbia 
had lower average scores than their 
eligible counterparts in central city 
schools. 

■ The average scale score for ineligible 
students at grade 4 in the District of 
Columbia was lower than the national 
average score in centred city^ schools. 
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Results for Grade 8 Public Schools 

At grade 8, the percentages of eligible 
students ranged from 67 percent in the 
District of Columbia to 84 percent in 
Chicago. Because the available data for 
eligibility for eighth-graders in Los Angeles 
did not meet reporting standards, no infor- 
mation related to eligibility is reported for 
this segment of the sample. 

■ Students at grade 8 who were not eligible 
for free/ reduced-price lunch had a higher 
average score than eligible students in 
every district where the data were suffi- 
ciently reliable for significance testing. 

■ At grade 8, both those students eligible 
for free/ reduced-price lunch and those 
not eligible in Atlanta and the District of 
Columbia had lower average scores than 
their counterparts in the central city 
public schools. 

Parents' Highest Level 
of Education 

Eighth-girade students who participated in 
the Trial Urban District Assessment were 
asked to indicate the highest level of educa- 
tion their p^irents had completed. Five 
response options — did not finish high 
school, graduated from high school, some 
education after high school, graduated from 
college, or 'T don’t know” — ^were offered. 

■ In all five districts, lower percentages of 
students reported that their parents hvid 
graduated from college than in the 
national sample. The percentages of 
students who reported that their parents 
did not graduate from high school were 
higher than the nation in Chicago, FIous- 
ton, and Los Angeles. 
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• Atlaiata and the District of Columbia had 
the highest percentages of students who 
reported that at le;ist one parent h-ad 
graduated from college (35 and 37 
percent, respectively). These percentages 
were significantly higher than those in 
Houston and Los iVngeles. Atlanta ;ilso 
had a higher percentage of students 
reporting parents with some education 
after high school than all the other 
districts. Houston ^ind Los Angeles had 
the highest percentages of students 
reporting parenl:s who did not finish high 
school (22 and 18 percent, respectively). 

• Average scores in all dismcts except 
Houston were lower for students who 
reported a college graduate parent than 
the nationd average score for their 
counterparts in the central city public 
schools. 

■ In Chicago and Houston, no statistically 
sigiiificant difference was detected 
between the average score of students 
with parents who did not finish high 
school and the average score of their 
counterparts in the central city schools, 
while the average score of these students 
in the Atlanta, the Distinct of Columbia, 
and Los Angeles was lower than the 
nation^il average score in the central city 
schools. 
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Introduction 




Overview of the Trial Urban District Assessment 
in Writing 2002 

Writing is an important tool for communicating ideas in 
school, work, and the community. Therefore, writing has 
always been central to elementary and secondary school 
curricula. The National Assessment of Educational Progress 
(NAE!.P) 2002 Writing Trial Urban District .Assessment 
(rUDA) initiates a national endeavor to provide the public 
with reliable evidence about urban fourth- and eighth-grade 
students’ achievements in writing, and makes comparisons 
to public schools nationally and central city schools.^ 

Brief History of the National Assessment 
of Educational Progress 

For more than thirty years, NAlEP has been providing 
reliable information about American students’ achievement. 

The purpose of NAEP reports is to inform educators, 
policy makers, parents, and the public. In order to do so, 

NAEP (also known as the Nation’s Report Card) regularly 
and systematically collects, analyzes, and reports valid and 
reliable information about what American students Imow 
and can do in a variety of subject areas. N AEP assesses 
representative national samples of fourth-, eighth-, and 
twelfdi-grade students, and representative samples of 
students in a number of states and other jurisdictions, 
including the District of Columbia and die Department of 
Defense schools (domestic and overseas). 

‘ “Ceiilnil city^^ is dermed iii chapter 2 and more completely hi die “Type of Location’^ 
section of appendix C. Central city includes nationally representative public schools 
located in central cities within inetropoUtan statistical areas as defined by the Federal 
Office of Management :uid Budget. It is not synonymous with “inner city.’^ 
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Authori 2 ed by Congress in 1969, NAEP 
is a continuing expression of the nation’s 
concern with students’ learning. The 
National Center for Education Statistics 
(NCES), one of three centers within the 
U.S. Department of Education’s Institute 
of Education Sciences (lES), manages 
NAEP. The National Assessment 
Governing Board (NAGB), an independent 
body, provides policy guidance for NAEP. 

NAEP does not report results for 
individual students or schools. In fact, for 
its first two decades, NAEP reported 
results only for the nation as a whole and 
for subgroups within the nation (e.g., for 
female students and male students). 

In 1988, Congress authorized a trial of 
state-level assessments. In 1990, NAEP 
undertook its first state-level assessment, 
of the mathematics performance of a 
representative sample of students in 
participating states. This successful trial of 
state-level assessments led to a broadening 
of the endeavor, and by 1994, NAEP was 
reporting to many states on their students’ 
performance in mathematics, reading, 
writing, and science. In 2002, NAEP 
assessed representative samples of students 
in the nation, in most of the states, and 
also in several U.S. jurisdictions. In 
addition, in 2002 NAEP assessed the 
performance of public-school students in a 
set of school districts. This report is the 
initial presentation of writing assessment 
results for each of five participating urban 
school districts and the District of 
Columbia. 

The national-, state-, and district-level 
reports of the NAEP 2002 assessment are 
available on the NAEP web site (http:// 
nces.ed.gov/ nationsreportcard/writing/) . 



^ No Child Behind Act of 2001, Pub. L. No. 107- 



The national- and state-level reports 
present nationwide results for grades 4, 8, 
and 12, and results for grades 4 and 8 
within the states and other jurisdictions 
that participated in the state-level 
assessment. The national and state reports 
also compare the results to the results of 
previous NAEP reading assessments. 

Background of the NAEP Trial 
Urban District Assessment 
in 2002 

Over the years, various constituents have 
requested reports on individual school 
results and district results, but the NAEP 
samples have not been designed to permit 
reporting of reliable data below the state 
level. The District of Columbia is an 
exception. The NAEP legislation (as well 
as other federal education programs) 
defines the District of Columbia as a state 
and authorizes its participation in state 
NAEP. Through its participation in state 
NAEP, student achievement data for the 
District of Columbia school system are 
regularly reported as part of the NAEP 
program. 

Federal appropriations authorized for the 
No Child Left Behind Act supported a 
multi-year study of the feasibility of a Trial 
Urban District Assessment as a component 
of the National Assessment of Educational 
Progress.^ Some large urban school districts, 
such as Los Angeles and New York, have 
enough students to meet NAEP 
requirements for sample size in reporting. 

In 2001, after discussion among the 
National Assessment Governing Board, the 
National Center for Educational Statistics, 
and the leadership of the Council of the 
Great City Schools, NAGB passed a 
resolution approving the selection of five 
large urban districts for participation in a 

-no, 115 Stilt. 1425 (2002). 



CHAPTER 1 • NAEP 2002 WRITING TRIAL URBAN DISTRICT ASSESSMENT 



Trial Urban District Assessment, a special 
project within NAEP.^ 

The trial assessment called for larger- 
than-usual samples within the districts, 
making reliable district-level data possible. 
The assessment allows these districts to 
make valid comparisons among themselves 
for the first time. Although individual 
states’ own assessments may report data by 
district and school, their results are based 
on different scores, scales, and test designs; 
therefore, until now, districts have not been 
able to compare themselves to districts in 
other states. 

By undertaking the Trial Urban District 
Assessment in reading and writing in 2002, 
NAEP continues a tradition of carefully 
extending its service to education, while 
preserving the rigorous sampling, scoring, 
and reporting procedures that have 
characterized the national and state 
assessments. The samples were large 
enough to provide data on subgroups 
within the districts, such as female students 
or Hispanic students. Thus, the 2002 data 
can serve the districts as a benchmark for 
studying changes in the performance of all 
their students and of particular subgroups 
of students. 

In addition to assessing subject-area 
performance of groups and subgroups, 
NAEP gathers contextual data about in- 
and out-of-school experiences and 
socioeconomic factors from background 
questionnaires given to students, teachers, 
and school administrators. Large amounts 
of additional data not included in this 
report are available on the NAEP web site 
(http:/ / nces.ed.gov/ nationsreportcard/ 
naepdata/). 



Selection of Urban Districts 

NAGB staff, assisted by representatives of 
the Council of the Great City Schools, 
identified five distncts for the trial 
assessment. Districts were selected that 
permitted testing the feasibility of 
conducting NAEP over a range of 
characteristics, such as district size, 
minority concentrations, federal program 
participation, poverty, and percentages of 
students with disabilities and limited 
English proficient students. This report 
presents data for the following participating 
urban districts: Atlanta City, Chicago 
School Distnct 299, Houston Independent 
School District Los ^\ngeles Unified, New 
York City Public Schools, and Washington, 
DC. All these p 2 irticipating districts are 
located in centiral cities. 

Overview of the NAEP 2002 
Writing Assessment 

Objectives and Content 
of the Assessment 

Each NAEP assessment has objectives 
described in a “framework” — a document 
that specifies the important content and 
process areas to be measured and the types 
of questions to be included in the 
assessment. NAGB directs a process for 
specifying these frameworks. 

The NAEP 1998 wnting framework is 
the blueprint that has specified the content 
and guided the development of the 1998 
and 2002 writing assessments.^ The 
framework establishes the assessment 
objectives and provides direction on the 
kinds of writing tasks to be included in the 
instrument. A task, or writing assessment 
item, is usually a short text or stimulus 



^ Nntioiial Assessmeut Governing Board. (2001, November 20). Minutes of the November 15—17, 2001 NAGB 
Meeting Committee Reports and Board Actions. Presented at die November 2001 NAGB meeting, Wasliington, DC. 

* National Assessment Governing Board. (1998). Framework and Specifications for the 199S National Assessment 

of Educational Progress. Wasliiiigton, DC: Aiidior. 
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posing a situation, concern, or topic about 
which students are asked to write under a 
stated time constraint. 

The framework resulted from a national 
process involving many parties concerned 
about writing education, including teachers, 
state education officials, subject-area 
specialists, researchers, and representatives 
of the general public. This effort was 
managed by the Center for Research 
on Evaluation, Standards, and Student 
Testing (CRESST), under the direction of 
NAGB. NAGB also contracted with ACT 
to develop detailed specifications for 
assessment items. 



The NAEP writing framework, informed 
by current research and theory, emphasizes 
that writing addresses a variety of purposes 
and audiences. The framework discusses 
three purposes for writing: narrative 
(telling a story), informative (informing the 
reader), and persuasive (persuading the 
reader). To ensure that NAEP writing 
assessments reflect the genres receiving the 
most instructional emphasis, the 
framework prescribes that NAEP writing 
tasks focus on these three purposes at 
grades 4, 8, and 12.^ Descriptions of 
narrative, informative, and persuasive 
writing appear in figure 1.1. 



Figure 1.1 Descriptions of the three purposes for writing in the NAEP writing assessment 



Purposes for Writing 



Narrative writing Norrative writing encourages writers to incorporate their imagination and creativity in the production of 
stories or personal essays. At its best, narrative writing fosters imagination, creativity, and speculation by 
allowing writers to express their thoughts and emotions, ond offers an opportunity for writers to analyze 
and understand their actions and those of others. 

The narrative prompts included in the NAEP 2002 writing assessment asked students to write many kinds 
of stories (most fiction, some nonfiction). Some of the prompts asked students to write in response to 
photographs, drawings, cartoons, poems, or stories (provided with the assessment). 

Informative writing In informative writing, the writer provides the reader with information. Informative writing may involve 
reporting on events or experiences or analyzing concepts and relationships. When used as a means of 
exploration, informative writing helps both the writer and the reader to learn new ideas and to reexamine 
old conclusions. 

informative prompts in the NAEP 2002 writing assessment asked students to write on specified subjects 
using many kinds of information, such as newspaper articles, charts, photographs, or reported dialogues 
(provided with the assessment), as well as their own knowledge. Students could write in a variety of 
formats, such as reports, newpaper articles, and letters. 



Persuasive writing Persuasive writing seeks to persuade the reader to toke action or to bring about change. This type of 
writing involves a cleor aworeness of what arguments might most offect the audience being oddressed. 
Writing persuasively also requires the use of such skills as analysis, inference, synthesis, and evaluatian. 

Persuosive prampts in the NAEP 2002 writing assessment asked students to write letters to the editor ar 
to friends, to refute arguments, ar to take sides in o debate. 



SOURCE Nntioiid Assassmeni Board. Wting Fwmemrk cndSpedficB^fv die / ^5 a/£^^ DC huhx. 



^ On the iinportauce of specifying purpose in writiug msfnicriou, see Oliver, E. (1989). Effects of Assignment on 
Writing Quality at Four Grade Levels. Em/ish Quarterly 2/(4), 224—32. 

Goniuui, S. (1995). \P't}uion<s into the Classroom: NAEP*s 1992 Portfolio Study. Wasliiiigton, DC: U.S. Department of 
Education, Office of Educational Research :iud Improvement, Natiouid Center for Education Statistics. 

National Assessment Governing Board. (1998). Writing Framework and Spedfuations for the 1998 National ^4.sscssmenl 
of Educational Progress. Wasbington, DC: Author. 
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As the framework notes, the purposes 
for writing are not always completely 
discrete. For example, a narrative essay may 
make a persuasive moral or ethical point, 
and a letter to an editor or congressional 
representative may include pertinent facts 
and information. In fact, many of the 
students whose writing received high 
ratings used integrated forms of 
presentation. The professional raters who 
evaluated the student responses were 
instructed not to penalize such blended 
presentations. 



The emphasis on each purpose for 
writing varies from grade to grade to match 
the differing levels of student development 
and instructional focus. The assessment 
emphasized narrative writing for fourth- 
graders, gave comparable weight to all 
three purposes for eighth-graders, and 
stressed persuasive writing for twelfth- 
graders. Table 1.1 shows both the 
percentage and actual number of writing 
tasks for each writing purpose at each grade 
level in the 2002 assessment. These 
distributions match the target percentages 
established by the framework. 



Table 1 .1 Distribution of writing tasks, by purpose for writing, in the NAEP 2002 writing assessment, 
grodes 4 and 8 





Percentage 


1 


Purpose far 


of writing tasks 


Actual number 


writing 


from framework 


of writing tosks 


Grode 






Narrative 


40 


8 


Informative 


35 


7 


Persuasive 


25 


5 


Grade 






Narrative 


35 


7 


Informative 


35 


7 


Persuasive 


30 


6 



SOURCE: U5. Departm^l of Educolhai, institule of Bfucotun Sciences, Center for EAicotion Statistics, f^ioncd Asesonem of Eckicotiofiot Progreu (HAEP), 2002 Writing Asesment. 
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In addition to specifying the percentage 
of the assessment that should be devoted 
to each writing purpose, the framework 
qualifies several elements of writing that 
should pertain to writing tasks across the 
assessment. These elements are considered 
by test developers in the construction of 
assessment items and are important for 
motivating student engagement in the 
assessment tasks. The 2002 assessment 
used many wnting tasks that specified the 
writer’s audience by asking students to 
wnte, for example, a letter to a friend or to 
a school board. Students also had 
opportunities to write in a variety of forms, 
such as essays, letters, reports, and stories. 
Wnting tasks may have used any of a 
variety of stimuli including photographs, 
cartoons, drawings, newspaper articles, 
letters, or literary works, such as poems or 
stones, to evoke written responses. While 
the limits of a timed assessment prevented 
students from engaging in the kinds of 
drafting and revising that can be done in a 
regular classroom environment, each 
writing task in the assessment included a 
planning page to encourage students in that 
phase of the writing process. In addition, 
every student received a bnef brochure 
with suggestions for planning and revising 
their writing. To meet the framework’s 
stated objective that students value writing 
as a communicative activity, background 
questions on the assessment asked students 
about their view of themselves as writers 
and their writing practices at home and at 
school. 



The 2002 Writing Assessment 
Procedure 

As the only federally authorized, ongoing 
nationwide assessment of student writing 
achievement, NAEP writing assessments 
must reflect the Ni\EP writing framework 
and expert perspectives on the 
measurement of writing competence. To 
that end, during the development process, 
the assessment undergoes stringent review 
by teachers, teacher educators, state 
officials, and measurement specialists. All 
components of the assessment are 
evaluated for curricular releviince, 
developmental appropriateness, fairness 
concerns, and adherence to the framework 
and test specifications. The 2002 writing 
assessment included twenty writing tasks 
calling for 25-minute responses at grades 4 
and 8. N^riEP writing tasks that have been 
released to die public, along with student 
performance data by state, are available on 
the N^AEP web site (http://nces.ed.gov/ 
nations rep o rt card / i tmrls /) , 

To minimize the burden on any one 
student, NAEP uses a procedure referred 
to as ''matrix sampling” in which an 
individual student at any given grade is 
administered only a small portion of the 
entire assessment. For example, at each 
grade, students received test booklets witli 
two writing tasks and two sections of 
questions regarding their home and school 
experiences related to writing achievement. 
Because a representative sample of 
students at each grade receives each 
assessment item, the results can then be 
combined to produce average group and 
subgroup results based on the entire 
assessment. In tofcil, the time required for 
each student to participate in the 2002 
N^\EP writing assessment was no more 
than 1 hour. 
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Procedures for Sampling 
Student Populations 

The N^\EP 2002 writing assessment was 
administered to fourth-, eighth-, and 
twelfth -graders at the national level and to. 
fourth- and eighth-graders at the state and 
district levels. NAEP uses a random 
selection process in order to obtain a 
representative sample of students for 
reporting national and state or jurisdiction 
results. Approximately 139,200 fourth- 
grade students in 5,500 schools; 118,500 
eighth-graders in 4,700 schools; and 18,500 
twelfth-graders in 700 schools were 
sampled and assessed. The public schools 
sampled also included charter schools, 
which in some cases were not managed by 
the urban school administration. 

The samples were selected using a two- 
stage sample design that selected schools 
within participating districts and then 
selected students within schools. In this 
report, all references to samples — national, 
central city, or district — include public- 
school populations only. 

Sampling for the Trial Urban District 
Assessment was modeled on the procedure 
for sampling states. The number of 
participating schools ranged from 38 to 76 
per district in the fourth grade and from 15 
to 69 per district at the eighth grade. The 
number of participating students per 
district ranged from 924 to 2,037 at the 
fourth grade and from 1,109 to 1,778 at the 
eighth grade. For information on sample 
sizes and participation rates by district, see 
“Urban District Samples” in appendix C. 

The overall participation rates for 
schools and students in the national, state, 
and urban district assessments must meet 
statistical guidelines established by NCES 



and NAGB in order for assessment results 
to be reported publicly. Data are not 
reported to the public for a state, 
jurisdiction, or urban district that 
participates but does not meet minimum 
participation rate guidelines. For more 
information about participation guidelines, 
see appendix C. Participation rates for the 
jurisdictions and urban districts were 
calculated the same way as rates were 
computed for the nation. 

NAEP endeavors to assess all students 
selected in the random sampling process, 
including students with disabilities (SD) 
and/or students who are classified by their 
schools as limited English proficient (LEP). 
The percentages of students classified as 
SD and/or LEP in all participating states 
and jurisdictions are available in an 
interactive database at the NAEP web site 
(http:/ / nces.ed.gov/ nationsreportcard/ 
naepdata/). Information on SD and LEP 
students for the urban districts appears in 
appendix A. Percentages of students 
identified as limited English proficient, 
particularly at grade 4, appear much higher 
in some districts (Houston and Los 
Angeles) than in the nation. 

It is important to note that, guided by 
the student’s Individualized Education 
Program (lEP), as well as eligibility for 
“section 504” services, school personnel 
make decisions as to whether students with 
disabilities should be included in the 
assessment.'" Results presented in this 
report include the performance of students 
with disabilities who can be meaningfully 
assessed without accommodations or with 
the kinds of accommodations NAEP 
provides, such as small-group settings, 
extended time, or the use of a scribe or 



* Section 504 ol tlie Rehabilitation Act of 197.3 is a civil rights law designed to prohibit discrimination on the basis 
of disability in programs and activities, including ecUicatioa, diat receive federal financial assistance. 



CHAPTER I • NAEP 2002 WRITING TRIAL URBAN DISTRICT ASSESSMENT 



computer for test administration. Schools 
also decide whether to include LEP 
students, based on NAEP’s guidelines. The 
guidelines ask them to judge the student’s 
ability to participate in the assessment in 
English and to consider the number of 
years the student has been receiving 
instruction in English. More information 
about types of testing accommodations is 
available in appendix A of this report. 

Percentages of students excluded from 
NAEP may vary considerably across 
states /districts and, within a state or 
district, from one year to another. For 
example, the percentage of fourth -grade 
students identified as having disabilities 
and or limited English proficiency ranged 
from 19 to 30 percent in public schools in 
the nation, in central cities, and in three of 
six districts. Atlanta was below the 
national average, with only 7 percent of its 
fourth-grade students identified in these 
groups, and Houston and Los Angeles were 
above die national average, with 40 and 52 
percent, respectively, of their fourth-grade 
students identified in these groups (in both 
of these cases the majority of the identified 
students had limited English proficiency). 
Given that miiny students identified in 
tJiese groups p;U‘ticipate in the assessment, 
the percentage of fourth-grade students 
who were excluded ranged between 7 and 
10 in public schools across the nation, in 
central cities, and in four of the six public 
school districts included in this report. In 
contrast to this, in Atlanta only 2 percent 
of the fourth -grade students were excluded 
from the assessment because they had 
disabilities (1 percent) or were limited 
English proficient (1 percent). At the same 



time, 15 percent of the fourth-grade 
students in Houston were excluded from 
the assessment, with 13 percent excluded 
because they had limited English 
proficiency. 

In the eighth grade, the percentage of 
students identified with disabilities iind/or 
limited English proficiency ranged from 8 
to 22 percent in public schools across the 
nation, in central cities, and two of the five 
districts. Atlanta was below the national 
average, with only 8 percent of its eighth- 
grade students identified in these groups; 
and Houston and Los Angeles were above 
the national average with 27 and 35 per- 
cent, respectively, of their eighth-grade 
students identified in these groups (in both 
of these cases the majority of the identified 
students had limited English proficiency). 
The percentages of eighth-grade students 
excluded from the assessment were more 
similar across jurisdictions. The exclusion 
rates for the nation, central city public 
schools, and four of the five districts 
ranged from 5 to 7 percent. In contrast, 
Atlanta excluded only 3 percent of its 
eighth-graders. 

In both grades the percentage of stu- 
dents assessed with accommodations 
ranged from 0 or 1 percent to 11 percent 
across jurisdictions. The variability in the 
identification, exclusion, and accommoda- 
tion rtes should be tiken into consideration 
in interpreting the results and making 
comparisons (see appendix A, table A.l). 
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Evaluating Studenfs"^ Writing 

Student responses in the NAEP 2002 
writing assessment were evaluated 
according to scoring guide criteria 
describing six performance levels. Scoring 
guides were developed for narrative, 
informative, and persuasive writing at each 
grade level. For example, the same scoring 
guide was used for all grade 8 narrative 
tasks. The guides included specific notes 
for raters that described various student 
approaches to the writing task and offered 
anchor or prototypical student responses at 
each grade level. For each task, a wide 
spectrum of student approaches was 
judged to be acceptable. Acknowledging 
developmental differences between fourth-, 
eighth-, and tu^elfth -graders, the scoring 
guides reflect higher performance 
expectations for higher-level students (for 
more details, see chapter 4 of the NAEP 
2002 writing report).^ Following the 
framework, the scoring guides emphasize 
students' abilities to develop and elaborate 
ideas, organize their thoughts, and write 
grammatically correct prose. The criteria 
for measuring command of written English 
mechanics differed by grade, but were the 
same across the three purposes for wnting 
(narrative, informative, and persuasive) 
within each grade. Since a time-controlled 
wnting context constrains students' 
opportunities to plan and revise, responses 
to assessment tasks were viewed as first 
drafts and evaluated accordingly. 



Student responses to all 25-minute tasks 
were analyzed based on descriptive rubrics 
in the 6-level scoring guides for narrative, 
informative, and persuasive writing. A scale 
of 1 through 6 — Unsatisfactory, 
Insufficient, Uneven, Sufficient, Skillful, 
and Excellent — was used in evaluating 
individual student responses to writing 
tasks. 

Reporting the Writing 
Assessment Results 

Results from the NAEP wnting assessment 
are presented in two ways: as scale scores 
on a scale from 0 to 300 and as percentages 
of students attaining achievement levels 
(Basic) Proftdenty and Advanced). The scale 
scores, indicating how much students know 
and can do in writing, ^ire presented as 
average scale scores and as scale scores at 
selected percentiles. In this report, the term 
average scores refers to average scores on the 
N^\EP writing scale. The achievement 
level results provide Eirther information by 
indicating the degree to which student 
performance meets the standards set for 
what they should know and be able to do. 
Results are reported only for groups or 
subgroups of students; an individual 
student's performiince cannot be reported 
based on NAEP assessment. 

Student responses to all 25-minute 
writing tasks were analyzed to determine 
the percentage of students sconng at each 
level on the 6-level guides for narrative, 
informative, and persuasive writing. The 
analysis entails summarizing the results on 



^ Petsky, H. R., Dnaiie, M. C., and Jin, Y (2003). The Natiotis Report Card: WriHng 2002. NCES 2003-529. Wasliiug- 
toii, DC: U S. Department ot Echicadon, lusdtiite of Ediicadou Sciences, National Center for Education 
Statistics. 
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separate subscales for each writing purpose 
and then combining the separate scales to 
form a single composite writing scale. This 
analysis yields the overall scale of 0—300 
for each grade (grades 4 and 8). 
Perform;ince for each grade is scaled 
separately; therefore, average scale scores 
cannot be compared across grades. Thus, 
equal scores on grade 4 iind grade 8 scales 
do not imply equal levels of writing ability. 
(For more information on sculling 
procedures, see “Data Analysis and Item 
Response ITeory (IRl^ Scaling'’ in 
appendix C of this report.) 

Student writing performance is also 
reported in terms of three achievement 
levels: Basic, Proficient, and A^dvanced. 

The three achievement levels are defined 
as follows: 

• Basic: lliis level denotes partial mastery 
of prerequisite knowledge and skills that 
are fundamental for proficient work at each 
grade. 

• Pmfitcient: This level represents solid 
academic performance for each grade 
assessed. Students reaching this level have 
demonstt''ated competency over challenging 
subject matter, including subject-matter 
knowledge, application of such knowledge 
to real-world situations, and analytical 
skills appropriate to the subject matter. 

• .Advanced This level signifies superior 
performance. 



For reporting purposes, achievement 
level cut scores iire placed on the writing 
scale to show the cutpoints between the 
Basic, Pf'oficient, and Advanced levels. The 
achievement level results are then report:ed 
as percentages of students within each 
achievement level range, as well as the 
percentage of students at or above Basic 
and at or above Proficient. 

The achievement levels are performance 
standards adopted by NAGB as part of its 
statutory responsibilities. The levels 
represent collective judgments of what 
students should know and be able to do for 
each grade assessed. They are based on 
recommendations from broadly 
representative panels of classroom 
teachers, education specialists, and 
members of the general public. As 
provided by law, the National Center for 
Education Statistics (NCES), upon review 
of congressionally mandated evaluations of 
NAEP, has determined that the 
achievement levels are to be used on a trial 
basis until it is determined that the 
achievement levels are “reasonable, valid, 
and informative to the public.”® However, 
both NCES and NAGB believe these 
performance standards are useful for 
understanding trends in student 
achievement. They have been widely used 
by national and state officials as a common 
yardstick of academic performance. For a 
full discussion of achievement levels, see 
“The Setting of Achievement Levels” and 
“Trial Status of Achievement Levels” 
in chapter 1 of the NAEP 2002 writing 
report.^ 



No Cliiia Left behiud Act of 2001, Pub. L. No. 107-110, 115 Stat. 1425 (2002). 

Persky, H. R., Daauc, M. C., aud Jin, Y. (2003). The Nation's Report Card: W'ntino 2002, NCES 2003-529. Wasliing- 
tou, DC: U. S. Depairtmeut of Education, Institute of Education Sciences, National Center for Education 
Statistics. 
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Specific definitions of the Basic, 

Vroficient, and Advanced writing achievement 
levels for grades 4 and 8 are presented in 
figures 1.2 and 1.3. The achievement levels 
are cumulative; therefore, students 
performing at the Proficient level also 
display the competencies associated with 
the Basic level, and students at the Advanced 
level also demonstrate the competencies 
associated with both the Basic and the 
Proficient levels. The score ranges for the 
NAGB achievement levels on the NAEP 
scale are as follows: Grade 4, Basic: 115— 
175; Proficient: 176—224; Advanced: 225 and 
above; Grade 8, Basic: 114—172; Proficient: 
173—223; Advanced: 224 and above. 

Interpreting NAEP Results 

The average scores and percentages 
presented in this report are estimates based 
on samples of students rather than on 
entire populations. Moreover, the collection 
of writing tasks used at each grade level is 
but a sample of the many tasks that could 
have been used to assess the skills and 
abilities described in the NAEP writing 
framework. The results are subject to a 
measure of uncertainty, reflected in the 
standard error of the estimates — a range of 
a few points plus or minus the average 
score or percentage — which accounts for 
potential fluctuations in average scores or 
percentages tliat are due to sampling and 
measurement error. The estimated standard 
errors for the estimated scale scores and 
percentages in this report are accessible 



through the N^AEP Data Tool on the 
NhAEP web site (http://nces.ed.gov/ 
nationsreportcard/ naepdata/). Examples 
of these estimated standard errors iire also 
provided in appendix C of this report. 

Ihe differences between scale scores and 
between percentages discussed in the 
following chapters take into account the 
standard errors associated with the 
estimates. Comparisons :ire based on 
statistical tests that consider both the 
magnitude of the difference between the 
group average scores or percentages and 
the standard errors of those statistics. 
Estimates based on sniiiller subgroups are 
likely to have relatively Large standard 
errors. As a consequence, some seemingly 
large differences may not be statistic;illy 
significant. That is, it cannot be determined 
whether these differences are due to the 
particular makeup of the samples of 
students who are selected or the true 
differences in the population of interest. 
Wlien this is the case, the term “apparent 
difference” is used in this report. 

Differences beuveen scores or between 
percentages are reported as such in this 
report only when they are significant from a 
statistical perspective. All differences 
reported are significant at the .05 level 
(with appropriate adjustments for multiple 
compmsons). The term “significant^’ is 
intended to identify statistically dependable 
differences in average scores or percentages 
and not to imply a judgment about the 
absolute magnitude or the educational 
relevance of the differences. Throughout 
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tdiis report, comparisons are made among 
districts and between districts and public 
schools in the nation, as well as between 
districts and centrd city public schools in 
the nation. (See ‘'NAEP Reporting 
Groups'’ in appendix C for details on how 
‘‘central city public schools” was defined.) 
Statisticiil rests are used to determine 
whether the differences be tureen average 
scores are significant. Only statistically 
significant differences are cited in this 
report. 

ITie reader is cautioned to rely on the 
reported differences in the text and tables, 
which iu*e statistically significant, rather 
than on the apparent magnitude of any 
differences. The standard errors are avail- 
able on the NAEP web site (http:/ / 

WWW. nces. ed .gov/ nationsre p ortcard / 
naepdata). 



Cautions in Interpretations 

As described earlier, the NAEP writing 
scale makes it possible to examine 
relationships between students’ 
performance and various background 
factors measured by NAEP; however, a 
relationship that exists between 
achievement and another variable does not 
reveal its underlying cause, which may be 
influenced by a number of other variables. 
Similarly, the assessments do not reflect the 
influence of unmeasured variables. The 
results are most useful when they are 
considered in combination with other 
knowledge about the student population 
and the educational system, such as trends 
in instruction, changes in the school-age 
population, and societal demands and 
expectations. 
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Figure 1 .2 Descriptions of NAEP writing achievement levels, grade 4 



Grade 4 

Achievement Levels 



The following statements describe the kinds of things fourth-grode students should be able to do in writing ot each level of achievement. These 
stotements should be interpreted with the constraints of the Notional Assessment of Educational Progress (NAEP) in mind. Student 
performances reported with respect to these descriptions ore in response to two oge-oppropriote writing tasks completed within 25 minutes each. 
Students ore not advised of the writing tasks in advance nor engaged in pre-writing instruction and preparation; however, they ore given a 
set of "ideas for planning and reviewing" their writing for the assessment. Although the Writing NAEP cannot fully assess students' abilities to 
produce o polished piece of writing, the results do provide valuable information about students' abilities to generate writing in response to a 
variety of purposes, tosks, ond audiences within o rather limited period of time. 



Bask Fourth-grade students performing ot the Basic level should be able to produce a somewhat organized and detailed 

( ] ] 5 j response within the time allowed that shows a general grasp of the writing task they have been assigned. 

Fourth-grade students performing at the ^as/c level should be able to produce a somewhat organized response within the time allowed 
that shows a general grasp of the writing task they have been assigned. Their writing should include some supporting details. Its 
grammar, spelling, and capitalization should be accurate enough to communicate to a reader, although there may be mistakes that get 
in the way of meaning. 



Proficient Fourth-grade students performing at the Proficient level should be able to produce an organized response within 
(176) allowed that shows an understanding of the writing task they have been assigned. Their writing should 

include details that support and develop their main idea, and it should show that these students are aware of the 
audience they are expected to address. 

Fourth-grade students performing at the /VoZ/rien/ level should be able to produce an organized response within the time allowed that 
shows an understanding of the writing task they have been assigned. Their writing should include details that support and develop the 
main idea of the piece, and its form, content, and languoge should show thot these students are aware of the audience they are expected 
to address. The grammar, spelling, and capitalization in the work should be accurate enough to communicate to a reader; there may 
be some mistakes, but these should not get in the way of meaning. 



Advanced Fourth-grade students performing ot the Advanced /eve/ should be able ta produce an effective, well developed 
(225) response within the time allowed that shows a clear understanding of the writing task they have been assigned 
and the audience they are expected to address. Their writing should include details and be dearly arganized, 
should use precise and varied language, and may show signs of analytical, evaluative, or creative thinking. 
Fourth-grade students performing at the Advanced level should be able to produce an effective, well developed response within the time 
allowed that shows a clear understanding of the writing task they have been assigned. Their writing should be clearly organized, making 
use of techniques such as consistency in topic or theme, sequencing, and a clearly marked beginning and ending. It should make use of 
precise and varied longuage ta speak to the audience the students are expected to address, and it should include details and elaboration 
that support and develop the main idea of the piece. Their writing may also show signs of analytical, evaluative, or creotive thinking. 
The grammar, spelling, and capitalization in the work should be accurate enough to communicote clearly; mistakes should be sa few and 
sa minor that a reader can easily skim over them. 

SOURCE: S. C Loomis and M. L Bourque (Eds. }.( 2001 ). HothndAss&smeai o/fi/i/ratet l^ogress/Vhievenieni Leveis, / 992- / 998 for Y/ntmg. Ui. Dept, of Eduration, Office of btucotionoi Research, Nationot 
Center for blucalion Statistics. Washington, OC. 
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Figure 1 .3 Descriptions of NAEP writing achievement levels, grade 8 



Grade 8 

Adiievement Levels 



The following statements describe the kinds of things eighth-grade students should be able to do in writing at each level of achievement. These 
statements should be interpreted with the constraints of the National Assessment of Educational Progress (NAEP) in mind. Student 
performances reported with respect to these descriptions ore in response to two age-appropriate writing tasks completed within 25 minutes 
each. Students are not advised of the writing tasks in advance nor engaged in pre-writing instruction and preparation; however, they are given 
a set of "ideas for planning and reviewing" their writing for the assessment. Although the Writing NAEP cannot fully assess students' abilities 
to produce a polished piece of writing, the results do provide valuable information about students' abilities to generate writing in response to 
a variety of purposes, tasks, and audiences within a rather limited period of time. 



Bask iighth^grade students performing at the Basic /eve/ should be able to produce an effective response within the 
|] ] 4 ] time aiiowed that shows a general understanding of the writing task they have been assigned. Their writing 
should show that these students are aware of the audience they are expected to address, and it should include 
supporting details in an organiied way. 

Eighth-grade students performing at the ^os/devel should be able to produce an effective response within the time allowed that shows 
a general understanding of the writing tosk they have been assigned. Their writing should show that these students are aware of the 
audience they are expected to address, and it should include supporting details in an organized way. The grammar, spelling, punctuation, 
and capitalization in the work should be accurate enough to communicate to a reader, although there may be mistakes that get in the 
way of meaning. 

Proficient Eighth-grade students performing at the Proficient level should be able to produce a detailed and organized 
(173) response within the time allowed that shows an understanding of both the writing task they have been assigned 
and the audience they are expected to address. Their writing should include precise language and varied sentence 
structure, and it may show analytical, evaluative, or creative thinking. 

Eighth-grade students performing at the Pro/ifc/en/ level should be able to produce an effective response within the time allowed that 
shows an understanding of both the writing task they have been assigned and the audience they are expected to address. Their writing 
should be organized, making use of techniques such as sequencing or a clearly marked beginning and ending, and it should make use 
of details and some elaboration to support and develop the main idea of the piece. Their writing should include precise language and 
some variety in sentence structure, and it may show analytical, evaluative, or creative thinking. The grammar, spelling, punctuation, and 
capitalization in the work should be accurate enough to communicate to a reader; there may be some errors, but these should not get 
In the woy of meaning. 



Advanced Eighth-grade students performing at the Advanced level should be able to produce o fully developed response 
(224) within the time allowed that shows a clear understanding of both the writing task they have been assigned and 
the audience they are expected to address. Their writing should show some analytical, evaluative, or creative 
thinking and may make use of literary strategies to darify a point. At the same time, the writing should be clearly, 
organized, demonstratmg predse word choice and varied sentence structure. 

Eighth-grade students performing at the Advancedk^el should be able to produce an effective and fully developed response within the 
time allowed that shows a clear understanding of both the writing task they have been assigned and the audience they are expected 
to address. Their writing should show some analytical, evaluative, or aeative thinking, and should demonstrate precise word choice and 
varied sentence structure. Their work should include details and elaboration that support and develop the main idea of the piece, and 
it may make use of strategies such os analogies, illustrations, examples, anecdotes, or figurative language to darify a point. At the same 
time, the writing should show that these students can keep their work clearly and consistently organized. Writing by eighth-grade 
students performing at the Advanced level should contain few errors in grammar, spelling, punctuation, capitalization, and sentence 
structure. These writers should demonstrate good control of these elements and may use them for stylistic effect in their work. 



SOURCE: S. C ioonn iHid M. L Bourque (Eds.). (200) }. HetiomdAssesmaJtofl^ucafhnaf Progress Aduevemeni Levds, / 992-} 998 fv Wr8ing. U.S. Dept, of Edurotmi, Office Biuoidonid Reseord), Hotiond 
Center for Educotion Stotistki Wudinigton, DC. 
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M Average Scale Score and 
^ Achievement Level Results for 

the Trial Urban District Assessment 

This chapter presents the NAEP 2002 writing results for the 
assessed urban districts and the District of Columbia at 
grades 4 and 8. Performance is reported as average scores on 
the N AEP writing scale, which ranges from 0 to 300, and in 
terms of the three writing achievement levels — Basic, Proficient 
and Advanced. 

Urban District Scale Score and Percentile Results 

Table 2.1 shows the overall performance of fourth- and 
eighth-grade students in the urban districts that participated 
in the 2002 writing assessment. In order to provide a context 
for these data, table 2.1 also displays the results for students 
attending public schools in the nation as a whole, as well as 
for public schools located in central cities across the nation. 

In table 2.1 and subsequent tables and figures in this report, a 
double asterisk (**) marks district statistics (average scores or 
percentages) that were found to be significantly different 
from the comparable statistic in public schools in the nation, 
and a single asterisk (*) marks district statistics that were 
significantly different from those of public schools in central 
cities. Following standards established by the Federal Office 
of Management and Budget, the U.S. Census Bureau defines 
a central city as a city of 50,000 people or more that is the 
largest in its metropolitan area, or can otherwise be regarded 
as “central,” taking into account such factors as commuting 
patterns. (See appendix C for more detailed definitions 
of geographical areas). The districts that participated in 
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the Trial Urban Assessment Atlvinta City, 
Chicago School District 299, Houston ISD, 
Los Angeles Unified, and New York City 
Public Schools, and Washington, DC are all 
located in central cities. The term means '"a 
city that is central”, not “the central part of 
a city” or the “inner city.” Central cities are 
defined for this report as including large ;md 
mid-size cities. ^\11 students in these districts, 
except for Houston and Los Angeles 
attended schools in centr;il cities. In the 
Houston and Los Angeles districts, some 
students included in this study attended 
schools located in the urban fringe. These 
included 5 percent and 19 percent of 
fourth-grade students in Houston '^md Los 
Angeles respectively, as well as 24 percent 
of eighth-grade students in Los ^Vngeles. 

The first column in the table presents the 
average score by district on the N^AEP 
writing scale. Results at each grade are 
scaled independently; therefore, cross -grade 
score comparisons cannot be made. The 
average score for fourth-grade students in 
public schools in the nation was 153 and for 
students in central city schools was 147. The 
average score for eighth-grade students in 
public schools in the nation was 152 and in 
central city schools was 143. At both grades, 
the average scores for students in the nation 
and for central city public schools generally 
were higher th;m those in these urb‘.in 
districts; exceptions to tliis generalization 
were found for New York City and Hous- 
ton. Fourth-grade students in New York 
City had higher average scores than fourth- 
graders in central city schools, while no 
significant differences were found between 
average scores for fourth- and eighth-grade 
students in Houston and those in central 
City schools. 



Details on the statistical significvince of 
these performance differences are provided 
below in the “Comparisons ^\mong Dis- 
tricts by Average Scale Scores” section. Data 
for New York City at grade 8 do not appear 
in this report because the district did not 
meet the required 70 percent participation 
rate (see appendix C, “Standards for Sample 
Participation and Reporting of Results”). 

"Ihe remaining columns in table 2.1 
indicate the scores attained on the 0-300 
writing scale of studeni:s at selected percen- 
tiles. Each percentile indicates the percent- 
age of students whose scores fell below a 
particul'.ir point on the NiVEP writing scale. 
For example, in Atlanta the 75th percentile 
score at grade 4 was 161 in 2002, indicating 
til at 75 percentof fourth-graders scored 
below 161. Table 2.1 shows the writing scale 
score for students scoring at the 25th, 50th, 
and 75th percentiles. An examination of 
scores at different percentiles at each grade 
indicates the score differences between 
higher-, middle-, and lower-performing 
students within a district .ind thus provides 
more detailed information than does the 
average score alone. The corresponding 
standard errors for these percentile scores 
are displayed in table C.6 in appendix C. 

When NAEP conducts a second Trial 
Urban District Assessment in writing, 
comparing the score values for percentiles 
across years will be useful in pinpointing 
where changes may have occurred within a 
district’s score distribution. For this report, 
the comparison of the scores at various 
percentiles across districts gives a rough 
indication of how score distributions 
may differ. 
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Toble 2.1 Averoge writing scoie scores ond selected percentiles, grodes 4 ond 8 public schools: 
By urbon district, 2002 







Scale score distribution 


1 




Average 
scale score 


25th percentile 


50th percentile 


75th percentile 


Grade 4 










Nation (PobRc) 


153 


128 


153 


178 


Central city (Public) ^ 


147 ** 


122 


146 


171 


Atlanta 


140 


117 *-** 


139 


161 *** 


Chicago 


138 


116 


137 


160 


District of Coiumbio 


135 


113 *-** 


134 


157 *** 


Houston 


148 


123 


147 


174 


Los Angeles 


141 


117 


141 


165 


New York City ^ 


153 " 


128 


154 ’ 


178 


Grade 8 










Notion (Public) 


152 


127 


153 


178 


Central city (Public) ^ 


143 


117 


144 


170 


Atlanta 


130 


107 


^ 2^ ^ 


151 


Chicago 


136 


111 


136 


160 


District of Columbia 


128 


105 


128 ’ 


152 


Houston 


138 


113 


139 


165 


Los Angeles 


128 


104 


128 ’ 


152 



^ Although doemed suffident reporting, the torget response rote spedhed m the NAEP guidelines wos not met. 

* Significantly different from centrol city public schools. 

Significontly different from notion (public schools). 

I For comparison, ot fourth grode 66 percent of students in centrol city public schools and 40 percent in public schools noHonolly were non^Whim. Also, 61 percent of students in centrol city public schools 
ond 43 percent in publk schools notionolty were eligible for free/reduced-price lundi. 

^ For comparison, at eighth grade 61 percent of students in central city public schoob ond 36 percent in public sdtoob notiondly w^e non-White. Abo, 48 percnit of students in centrol city public schoob 
and 34 percent in public schoob notiondly were eligible for free/reduced-prIce lunch. 

SOURCF: U.S. Department of Education, Institute of Education Sciences, Notisnol Center for Biucotion Stotistics, Kotiond Assessment of Educational Process (NAEP), 2002 Triol Urban Oistiict Writing 
Assessment and 2002 lotting Assessement. 
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Comparisons Among Districts 
by Average Scale Scores 

Figures 2.1 and 2.2 display the results of 
statistical significance tests of differences in 
NAEP 2002 average writing scores between 
any two urban districts at grade 4 and at 
grade 8, respectively. These figures are 
similar to mileage charts on travel maps. 
Read across the row corresponding to a 
district to the column of the jurisdiction 
chosen for comparison. If the resulting cell 
is not shaded, the difference between the 
two average scores was not found to be 
statistically significant. If the cell is lightly 
shaded (with an upward arrow), the average 
scale score of the district on the row on the 
left was higher than that of the jurisdiction 
named at the top of the column. The darkly 
shaded cells (with a downward arrow) 
indicate that the average scale score of the 
district on the rows on the left was lower 
than that of the jurisdiction selected at the 
top of the column. For example, in figure 
2.1, the leftmost cell in the fourth row 
compares the average score at grade 4 in 
Atianta to the average score in the nation. 
The shading and the arrow in this cell 
indicate that the average score in Adanta 
was lower than the national average score 
and that the difference was statistically 
significant. 



At grade 4, the average score for 
students in the nation was higher than the 
average score in each of the districts except 
New York City and Houston, where the 
average scores were not found to be 
significandy different from the national 
average score. Fourth-grade average scores 
were lower in most districts than the average 
score in the central city schools except in 
New York City, where the average score was 
higher, and in Houston, where the average 
score was not found to differ significandy. 
At grade 4, the average score for students in 
New York City was higher than those in all 
the other districts except Houston. The 
average score for students in Houston was 
higher than average scores in Adanta, 
Chicago, and the District of Columbia, but 
not found to differ significandy from 
average scores in Los Angeles and New 
York City. The average score for students in 
Los Angeles was higher than the average 
score in the District of Columbia, but not 
found to differ significandy from the 
average scores in Adanta and Chicago. 

Other apparent differences were not 
determined to be statistically significant. 
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Figure 2.1 Cross-district comparisons of average writing scale scores, grade 4 public schools: By urban district, 2002 



Grade 4 



Instructions: Read across the row corresponding to an urban district listed to the left of the chart. Match the shading intercity 
to the key below to determine whether the average writing scale score of this district was found to be higher than, not signifi* 
cantly different from, or lower than that of the entity named in the column heading. For example, in the row for Atlanta, 
Atlanta's average score was lower than the average scores in the nation, central cities, New York City and Houston, but not found 
to differ significantly from the average scores in Los Angeles, Chicago, ond the District of Columbia. 
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New York City^ 
Houston 
Los Angeles 
Atlanta 
Chicago 
District of Columbia 




District had higher 
average scole score 
thon the district 
listed ot the top of 
the figure. 



□ No significant 

difference detected 
from the district 
listed at the top of 
the figure. 



District had lower 
overoge scole score 
thon the district 
listed ot the top of 
the figure. 



^ Althou^ deemed sufficient for reporting, the target response rate speciNed in the NAEP guidelines mi not met. 

f for comporison, ot fourth grade ^ percent of students in centroi dty public schools ond 40 percent In public schools notionoliy were non-V^ite. Also, 61 percent of stud^ in central cHy public schools 
and 43 percent in public schools notiondly were eligible for free/red uted -price lunch. 

NOTE: The between-dhtrict comparisons take into account somphng and meosurernent error and that each district is being compared with every other district diown. Significance is determined by on 
opplicQtion of a multiple-comporison procedure. 

SOURCE: Ui. Deportment of Educotion, Institute of biucotion Scien^, Notional Center hir Mication Statistics, National Assessment of Educotionol Pro^ss (NAEP), 2002 Tried Urbon District Writing 
Assessment ond 2002 VAiting Assessement. 
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Figure 2.2 Cross’^district comparisons o{ average writing scale scores, grade 8 public schools: By urban district, 2002 



Grade 8 



Instructions: Read across the row corresponding to an urban district listed to the left of the chart. Match the shading 
intensity to the key below to determine whether the average writing scale score of this district wos found to be higher than, 
not significantly different from, or lower than the entity named in the column heading. For example, in the row for the 
District of Columbia, the average score for the District of Columbia was lower than the overage scores for the nation, central 
city, Houston and Chicago, and not found to differ significantly from Atlanta and Los Angeles. 
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Houston 
Chicago 
Atlanta 
District of Columbia 
Los Angeles 




Disfrict had higher 
^ average scale score 
than the district 
listed at the top of 
the figure. 

□ No significant 

difference detected 
from the district 
listed at the top of 
the figure. 



District had lower 
overage scale score 
than the district 
listed at the top of 
the figure. 



^ For (omporison, ot eighifi grade 6) percent of students in central city public schools and 36 percent in pubik schools nationolly «%re non-White. Also, 48 percent of students in central city public schools 
and 34 percent in puhik schools nationally were eligible for free/ reduced-price lunch. 

NOTE: The between-dlstrkt comporisons take into account sampling and measurement error and that each district is being compared wiHi every other district shown. Signrficonce Is de^mkied by on 
di^lication of o mulHpleKomparison procedure. 

SOURCE: U.S. Deporimeni of Education, Institute of Education Sciences, Notional Onter for Erhicotion Statistics, National Assessment of Educational Progress (NAEP), 2002 Trial Urbon DistrkI Writing 
Assessment and 2002 V/riting Assessement. 



As shown in figure 2 . 2 , eighth-grade 
students’ average scores in all five districts 
were lower th^in the national average and 
lower thiin the central city average, with the 
exception c:>f Houston, where the average 
score was not found to be significantly 
different from the average score for central 
city schools. At grade 8, the average score in 
Houston was higher than the average scores 



in Atlanta, the District of Columbia, ,ind 
Los Angeles. The average score in Chicago 
was higher than the average scores in die 
LOistrict of Columbia and Los Angeles and 
not found to differ significantly from the 
scores in Houston and Adanta. Other 
appvirent differences were not determined to 
be stadstically significant. 
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Writing Achievement 
Level Results 

III addition to reporting average writing 
scale scores, NAEP reports writing perfor- 
mance by achievement levels. The writing 
achievement levels 2 ire BasiCj 'Proficient and 
Advanced, The setting of the achievement 
levels is discussed in chapter 1. 



Table 2.2 presents the percentages of 
students at grades 4 and 8 who performed 
below the Basic level, at or above the Basic 
level, at or above the dent levels and at 
the Advanced level. 

At grade 4, the percentage of students 
performing at or above Profident ranged 
from 11 percent in the District of 
Columbia to 27 percent in New York City. 



Toble 2.2 Percentage of students at or above each writing achievement level, grades 4 and 8 public schools: 
By urban district, 2002 





Selow Bask 


At or above 
Bask 


At or above 
ProHimi 


At Advamed 


Grade 4 




Notion (Public) 


15 


85 


27 


2 


Central city (Public) ^ 


19 *. 


81 ** 


21 ** 


2 


Atlonto 


23 


77 *" 


13*'** 


1 


Chicago 


24 


76 *'** 


12*'** 


# 


District of Columbia 


27 


73*'** 


11*'** 


1 


Houston 


19 


81 


23 


2 


Los Angeles 


23 




16*'** 


1 


New York City ^ 


15 


85 


27 


2 


Grade 8 










Notion (Public) 


16 


84 


30 


2 


Central city (Public) ^ 


23*" 


77 ** 


22** 


1** 


Atlanta 


32 


68*'** 


10 


# 


Chicago 


28 


72*'** 


16*'** 


1 


District of Columbia 


34 *** 


66 *'** 


10*'** 


# 


Houston 


26** 


74 ** 


19** 


1 


Los Angeles 


36*'** 


64*'** 


11*'** 


# 



# Percentage rounds to zero. 

^ Although deemed suffident for reporting, the to^et response rote specified in the NAEP guidelines wos not met. 

“ Significantly different from central city public schools. 

Significantly (Afferent from nation (public schools}. 

^ For camporison, ot fourth grode 66 percent of students in central dty public schools and ^0 percent In puldic schools nationally were non.Vifhite. Also, 61 percent of stud»tts lit central city public schools 
and 43 percent in public schools nationally were eligible for free/reduced-fnice htnch. 

^ For camporisan, at eighth grade 61 percent of students in centrol city public schools and 36 percem in pidthc schools nationally vn^e non-Vl^ta. Also, 48 percent of students in central city public schools 
and 34 percent in ptdthc schools notionolly were eligible for free/reducedfrice lunch. 

NOTE: Percentages below ond at or above dos/ir may not odd to 1 00, due to rounding. 

SOURCE: U.S. Department of Education, Inshtute of Education Sciences, Notional Center for Education Slotisiia, NatQo||&essment of Educatlanol Progress (NAEP), 2002 Trial Urbon District Writing 
Assessment and 2002 WHting Assessement. D ^ 
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Twenty-seven percent of the students across 
the nation performed at or above the 
Pr(^de/it level, while 21 percent of all central 
city students reached this level of perfor- 
mance. The percentages of fourth-grade 
students reaching at least the Bask achieve- 
ment level ranged from 73 percent in the 
District of Columbia to 85 percent in New 
York City. In the nation as a whole, 85 
percent of students and 81 percent of 
centoil-city students performed at or above 
the Bask level. 

At grade 8, the percentages of students at 
or above Proficient n?a\ged from 10 percent in 
Atlanta and the District of Columbia to 19 
percent in Houston, while 30 percent ot 
students in public schools in the natic)n and 
22 percent in central city public schools 
performed at or abcwe the Profident level. At 
grade 8, the percentages of students per- 
forming at or aboA^e Bask ranged from 64 
percent in Los ^\ngeles to 74 percent in 
Houston. In the national public schools, 84 
percent of students performed at or above 
Bask, and 77 percent of central city students 
performed at the same level. 



Comparisons Among Districts 
by Achievement Levels 

Figures 2.3 and 2.4 are “mileage charts'’ 
similar to figures 2.1 and 2.2; in this case the 
data compared represent percentages of 
students performing at or above the Prf- 
dent level. At grade 4, the percentages of 
students performing at or above Profuient kx 
public schools nationdly and in public 
central city schools were higher than the 
percentages in iill districts except Houston 
and New York City, where tlie differences 
were not found to be statistically significant. 
At grade 4, the percentage of students 
performing at or above Prof dent in New 
York City was higher than the percentages 
in four other districts and not found to 
differ significantly from the percent:age in 
Houston. 

Vthile not displayed in the figures, there 
were also interesting patterns in the percent- 
ages of students performing at or above th6 
Bask level. No significant difference was 
found betv^^een the percentages of fourth- 
grade students performing at or above Bask 
in Houston and New York City and the 
percentages of fourtli-graders performing at 
the same level in the nation and central 
cities. The percentages of fourth -grade 
students performing at or above Bask were 
lower in Chicago and the District of Co- 
lumbia than the percentages of students 
performing at or above Bask in central cities. 
Fourth -grade students in Houston and New 
York City were more likely than those in the 
District of Columbia to perform at or 
above the Bask level. Other apparent differ- 
ences were not found to be statistically 
significant. 
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Figure 2.3 Cross-district comparisons of percentage of students at or above Proficient m writing, grade 4 public schools: 
By urbon district, 2002 



Grade 4 



Instructions: Read across the row corresponding to an urban dbtrict listed to the left of the chart. Match the shading 
intensity to the key below to determine whether the percentage of students at or above Proficient in this district was found to 
be higher than, not significantly different from, or lower than the district or jurisdiction in the column heading. For example, 
in the row for Chicago: Ihe percentage of students at or above Proficient in Chicago was lower than the percentage in the 
nation, central cities. New York Gty, and Houston, but not found to differ significantly from the percentage in Los Angeles, 
Atlanta, and the District of Columbia. 
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New York City^ 
Houston 
Los Angeles 
Atlanta 
Chicago 
District of Columbia 







District had higher 
percentage than the 
district listed ot the 
top ot the figure. 



□ No significant 

difference detected 
from the district 
listed ot the top of 
the figure. 



n District had lower 
percentage than the 
district listed ot the 
top of the figure. 



^ Although deemed sufficient for reporting, the target response rote specified in the NAEP guuklines was not met. 

^ For comparison, ot fourth grade 66 percent of students in centroi dty public schools and 40 percent in public Hhooh notiondly were non-White. Also, 61 percent of students in rantrol city public schools 
ond 43 percent in public schools notlonolly were eiigilde for free/ reduced-price lundi. 

NOTE; Tfie between-district comparisons take Into aaount sompllng ond meouirement error and ihai each chstrict is being compared with every other district shown. Significonce is determined by on 
application of a muhipie-comporison procedure. 

SOURCE: U.S. Deportment of Educotion, Institute of Education Sciences, Notionol Center for Education Statistics, NaHond Assessment of Educoticmal Progress (NAEP), 2002 Trio! Urban District Writing 
Assessment and 2002 Writing Assessement. 
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At grade 8, rfie percentage of students 
performing at or above Proficient \v?iS higher 
for tlie nation than for all urban districts 
assessed, ;ind higher for central cities than 
for all urban districts assessed except Hous- 
ton. No significant difference was detected 
between the percentages of students per- 
forming at or above Proficient in Houston 
and central cities. The percentages of 
eighth-graders performing at or above 
Proficient in Chicago and Houston were not 
found to differ significantly from each 
otlier, cuid both were higher thiin the com- 
parable percentages in the District of 
Columbia and Atlanta. 



The percentage of students performing 
at or above Basic higher in public 
schools in the nation tlian in all five districts 
reported and higher in the central city 
schools tlian in all these districts except 
Houston, which did not differ signific;intly 
from the central cities on this performance 
measure. Students in Houston were more 
likely than those in the District of Columbia 
and Los iVngeles to perform at or above the 
Basic level. 



Figure 2.4 Cross-district comparisons of percentage of students ot or above Proficient \n writing, grode 8 public schools: 
By urban district, 2002 



Grade 8 



Instructions; Read across the row corresponding to an urban district listed to the left of the chart. Match the shading I 
intensity to the key below to determine whether the percentage of students at or above Proficient in this district was found to " ■ 
be higher than, not significantly different from, or lower than the district or jurisdiction in the column heading. For example, 
in the row for Houston: The percentage of students at or above Proficient \n Houston was lower than in the nation, was not 
found to differ from the central dties and Chicago, and was higher than Los Angeles, the District of Columbia, and Atlanta. 
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Houston 
Chicago 
Los Angeles 
District of Columbia 
Atlanta 




Disfrict had higher 
percentage than the 
district listed at the 
top of the figure. 

□ No significant 

difference detected 
from the district 
listed ot the top of 
the figure. 






District hod lower 
percentage than the 
district listed at the 
top of the figure. 



^ For (omptirison, at eighth grade 61 percent of students in central city public schools end 36 percent in public sdtools noliondly \^re non-White, dso, 48 percent of students in central ciiy public schools 
ond 34 percent in public schools nationally were eligible for free/reduted-price lunch. 

NOTE: The between-distritt com{»irisons toko into onount somphng and measurement e^or ond that each dhtnet is being com]»ired mth ev^ other district shown. %iificance is determined by on 
application of a multipleKomponson procedure. 

SOURCE: U.S. Deportment of biucotion, institute of Educohon Sciences, Notionol Center for Education Statistics, National Assessment of Educotionol Process (NAEP), 2002 Trial U^an District Writing 
Assessment ond 2002 Wnling Msessement. 
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Results for Subgroups 

In addition to reporting on the performance of all students, 
NAEP provides results for a variety of subgroups of 
students for each grade level assessed. The subgroup results 
show how these groups of students performed in 
comparison with one another. When additional years of 
urban district assessments are conducted, these results will 
provide a basis for examining each group’s progress over 
time. 

Tliis chapter includes average writing scores and 
achievement level results at grade 4 for subgroups of 
students in six urban districts — the five that participated in 
the Trial Urban District Assessment and the District of 
Columbia — and results for grade 8 in Atlanta, Chicago, 
Houston, Los Angeles, and the District of Columbia. New 
York City data for grade 8 are not reported because 
participation did not meet reporting criteria. Results are 
reported by gender, race/ethnicity, students’ eligibility for 
free/ reduced-price lunch, and by parents’ highest level of 
education. 
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Throughout this chapter, student sub- 
group results in each district are also com- 
pared to the results for central city schools. 
In most cases, the average scores and 
achievement level results for central city 
schools were below those for the nation. 

All differences noted as such in this chapter 
are statistically significant. In interpreting 
the results, it is impornmt to bear in mind 
that the estimated average score for a 
particular group does not include the whole 
range of perform'^mce within that group. 
Differences in subgroup performance 
cannot be ascribed solely to students’ 
membership in an identified subgroup. 
Average student performance is affected by 
the interaction of a complex set of fiictors 
not discussed in this report or addressed by 
NAEP assessments. 

Performance of Selected 
Subgroups in the Trial Urban 
District Assessment 

Gender 

Educators and government agencies have 
produced a body of research rich in data 
documenting gender differences in liinguage 
arts achievement.^ National results for tlie 
NAEP 1998 and 2002 writing assessments 
indicate that female students outperformed 
male students ;at grades 4, 8, and 12. 



Table 3.1 shows average writing scores 
and achievement level results by gender at 
grade 4. Female fourtli-graders had a higher 
average score than male fourth -graders in 
each of the urbtin districts. Average scores 
for male students at grade 4 ranged from 
127 in the District of Columbia to 145 in 
New York City. No significant difference 
was detected between the average scores for 
male fourtli-grade students in Houston iond 
New York City and their counterp^urts in 
central city schools. The average scores of 
male fourtli-graders in Atlanta, Chicago, the 
District of Columbia, and Los i:\ngeles were 
lower than die average score for male fourdi- 
grade students in central city schools. 

The average scores for female students at 
grade 4 in the six districts ranged from 143 
in the District of Columbia to 160 in New 
York City. No significant difference was 
detected beKveen the average scores for 
female fourth -grade students in Houston 
and New York City and the average score 
for female fourth -graders in the centnil city 
schools. Female fourth-graders in Atlanta, 
Chicago, die District of Columbia, and Los 
Angeles had lower average scores than 
female fourth-grade students in central city 
schools. 



^ U.S. Dep sir linen I: of E due a lion. (2001). Reading for Understanding: Totuanis an R D Program in Reading Comprehension. 
Wasliingtou, DC: Author. 

MacMilhm, R (2000). Simultaneous Measurement of Reading Growth, Gender, and Relative- Age Effects: Many 
Faceted Riisch AppUed to CBM Reading Scores. Journal of Applied Measurement^ /(4), 393-408. 

Peterson, S. (2001). Gender Identities juid Self-Expression in Classroom Narrative Writing. Language Arts, 75(5), 
451-439. 

Thomas, P. (1994). Writing, Reailhig, and Gender. Gifted Education International, P(3), 154—158. 
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Table 3J Average writing scale scores and percentage of students at or above each achievement level, by gender, 
grade 4 public schools: By urban district, 2002 



Grade 4 



Mole 



Female 





Percentage 
of students 


Average 
stole score 


Below Bask 


At or above 
Bask 


At or above 
Profitieaf 


Advamed 


Nation (Public) 


51 


144 


20 


80 


18 


1 


Central city (Public) ^ 


51 


139 "" 


24 ** 


76 ** 


15** 


1 


Atlanta 


47 


131 


31 *'** 


69*'** 


8*'** 


1 


Chicago 


50 


130^*" 


32 *'** 


68*'** 


j 


# 


District of Columbia 


49 


127 


37 *-** 


63 *'** 


j * 


#*^** 


Houston 


48 


141 


24 


76 


17 


1 


Los Angeles 


51 


134 


29 ** 


71 ** 


11 *'** 


# 


New York Gty ^ 


50 


145 


21 


79 


21 


1 


Nation (Public) 


49 


162 


10 


90 


35 


3 


Central city (Public) ^ 


49 


155** 


13** 


87** 


27 ** 


2** 


Atlanta 


53 


147 


17** 


83 ** 


18 *'** 


2 


Chicago 


50 


145 *'** 


17** 


83** 


16*'** 


# 


District of Columbia 


51 


143 


18*'** 


82 


15*'** 




Houston 


52 


154** 


15** 


85** 


28 ** 


3 


Los Angeles 


49 


148*'** 


16** 


84 ** 


21 *'** 


]*** 


New York Gty ^ 


50 


160 


10 


90 


33 


3 



# Percentoge rounds to zero. 

^ Allfiough deemed sufficient for reporting^ the target response rote specified in the NAEP guicfeTmes wos not met. 

* Significontiy different from central city puhlit schools. 

Significontiy different from notion (public schoob). 

^ For comparison, ot fourth grode 66 percent of students in central city public schools and 40 percent in public schoob nation oily were ncm-V/hite. Abo, 61 percent of students in central city public schools 
ond 43 percent in public schools nationally were eligible for free/reduced-price lunch. 

NOTE; Percenmges below ond ot or obove Bam may not odd to 100, due to rounding. 

SOURCE; U.S. Department of Educotion, Institute of Education Sciences, Notionoi Center for Educotion Stotistks, Notionol Assessment of Educotlonol Progress (NAEP), 2002 Trial Urban District Writing 
Assessment ond 2002 Writing Assessement. 
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Table 3.2 Averoge writing scale scores and percentage of students at or above each achievement level, by gender, 
grade 8 public schools: By urbon district, 2002 



Grade 8 





Percentage 


Average 


i 


of students 


scale score 


Notion (Public) 


50 


141 


Centro! dty (Public) ^ 


50 


133’^" 


Atlanta 


49 




Chicago 


48 


126 ’** 


District of Columbia 


49 


120*'** 


Houston 


47 


124*'** 


Los Angeles 


50 


121 *'** 


Notion (Public) 


50 


162 


Central city (Public) * 


50 


153** 


Atlonto 


51 


137 *'** 


Chicago 


52 


145 *'** 


District of Columbia 


51 


136*'** 


Houston 


53 


150** 


Los Angeles 


50 


134*'** 



Below Bask 


At or above 

Bask 


At or above 
Profitieaf 


Advamed 


23 


77 


20 


1 


30 ** 


70** 


14** 


1 


40 *'** 


60*'** 




# 


37 ** 


63** 


11 1* 


1 


43 *'** 


57*'** 




# 


37 ** 


63** 




# 


42 


58*'** 


6*'** 


# 



9 


91 


40 


3 


15** 


85** 


30 ** 


2** 


24 


76*'** 


13*'** 


1 


19** 


81 ** 


21 *'** 


2 


25*'** 


75*'** 


14*'** 


# 


16** 


84** 


28** 


1 




71 *'** 


15*'** 


# 



^ Pertentaga rounds io zero. 

" SfgniBrontiy different from cenn'o) city ptdtilc schools. 

** Significmitly different ^om notion (puldic schools]. 

^ For comparison, ot eighth grade 61 percent of students in cenO'd city public schools and 36 percent in public schoob notiono% were non-White. Also, 48 percent of students in centrd city public schoob 
and 34 penent m pdtbc schoob nationally were eligible for ^ee/reduc«ff lice lunch. 

NOFE: Percentages below ond at ar above Ats/r may not odd to 100, due to rounding. 

SOURCE U.S. D^tortinent of Educotion, Institute of Education Sdences, National Cent^ for Education Statistics, National Assessment of Educational Pragress (NAEP), 2002 Trial Urisan District Writing 
Assessnvnt and 2002 Writing Assessement. 



Table 3.2 displays average scores and 
achievement level results by gender for 
grade 8. In each of the five districts re- 
ported, tJie average score of female students 
was higher than die average score of male 
students. Average scores for miile eighth- 
grade students ranged from 120 in the 
District of Columbia to 126 in Chicago. 

The average score for eighth-grade male 
students in central city scliools was higher 
tJiiin that in each of the five urban districts 
reported. 



For femiile students at grade 8 in the five 
districts, average scores ranged from 134 in 
Los Angeles to 150 in Houston. For female 
eighth-graders, die average score was higher 
in central city- schools thvin in any of die 
districts except Houston: no significant 
difference was detected betw^een the average 
score for female eighth-graders in Houston 
and in the central city^ schools. 
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An additional way to compare the 
performance of subgroups of students is to 
focus on the si 2 e of the difference between 
the average scores for the subgroups. Where 
the average scores of one subgroup have 
been consistently lower than those of 
another over time, the differences have 
come to be called ''gaps.” Figure 3.1 
presents the ‘'gaps” in average writing 
scores between female and male students in 
each district, in the central city schools, and 
in the nation. At grade 4, the gaps between 



the average scores of female and male 
students in the trial urban districts were not 
found to differ signific^uitly from 
compiurable gaps in public schools in the 
nation and in central cities. At grade 8, the 
score gap between female and male students 
was narrower in the District of Columbia 
than in public schools in the nation. In Los 
Angeles, the gender score gap was narrower 
than the comparable gaps in public schools 
in both central cites and in the nation. 



Figure 3J Gaps in average writing scale scores, by gender, grades 4 and 8 public schools: By urban district, 2002 



MewYorkGly 



Female average score minus Male average score 

Grade 4 



Nalion (Public) 
Central clly (Public) 
Atlanto 
Chicogo 
District of Columbio 
Houston 




Female average score minus Male average score 

Grade 8 




Score gaps 



Score gaps 



^ Although deemed sufficient for reporting, the target re^onse rote specified m the NAEP guidelines was not met. 

* SigniHcantly different from ^irol city public schools. 

** Significontly different from nation (public sdioofs). 

^ For comparison, ot fourth grode 66 percent of students In central city public schools and 40 percent in public schools notionolly were non-V/hite. Also, 61 percent of students in centrd city public schools 
ond 43 percent in public schools notionoliy were eligible for free/reduced-price lundi. 

^ For comparison, ot eighth grode 61 percent of students in centrol city public schools and 36 percent in public schools notlondly were non-White. Also, 48 percent of students in centrd city public schools 
ond 34 percent in public schools notlondly were etigftle for free/ reduced-price lundi. 

NOTE: Score gops ore colculoted bosed on differences between unraunded overage scole scores. 

SOURCE Ui. Deportment of Elucotion, Institute of Bhicotion Sciences, Notional Center for Bfucotion Stotistlcs, Notiond Assessment of Elucotiono! fVogress (NAEP), 2{H)2 Trial Urbon District Writing 
Assessment and 2002 Writing Assessement. 
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Race/Ethnicity 

For the purpose of smdying the progress of 
subgroups, NAEP collects information 
from school records on the racial/ ethnic 
identification that best describes each 
participating student, llie six mutually 
exclusive categories are VCTnite, Black, 
Flispanic, Asian/Pacific Islander, American 
Indian (including Alaska Native), and Other. 
This information was the primary 
contributor to the classifications appearing 
below. For further details, see ‘‘NAEP 
Reporting Groups^’ in appendix C. 

Tables 3.3 and 3.4 show average scores 
and achievement level results by racial and 
ethnic group membership for public-school 
students in the urban districts at grades 4 
and 8, respectively. Only the race/ethnicity 
categories with sufficient membership to 
meet reporting requirements in the urban 
districts are reported below. 

The distribution of students in terms of 
race /ethnicity in the urban districts differs 
from the distribution in the nation’s public 
schools. Black or Hispanic students 
constitute the majority or the largest 
fraction of students in each of the urban 
districts in the trial assessment. Black 
students constitute over 85 percent in 



Atlanta and in the District of Columbia at 
fourth and eighth grades, as well as 47 
percent of Chicago’s fourth grade and half 
of its eighth grade. Hispanic students 
constitute a majority at fourth and eighth 
grades in Los Angeles and at eighth grade in 
Houston, as well as 47 percent of the fourth 
grade in Houston and 43 percent of the 
fourth grade in New York City. The highest 
percentage of White students in the urban 
districts is 15 percent in New York at grade 
4 and 11 percent in Chicago at grade 8, 
whereas in the national writing assessment 
White students constitute majorities at both 
fourth and eighth grades: 60 percent at 
grade 4 and 64 percent at grade 8. 

In five of the six urban districts in which 
a reliable comparison could be made. White 
fourth-graders had higher average scores 
than their Black and Hispanic counterparts. 
The White /Hispanic comparison in Atlanta 
could not be tested for statistical 
significance due to insufficient sample size. 
No significant difference was detected 
between the average scores of Asian 
American/Pacific Islander students and 
their White counterparts in Los Angeles and 
New York City. 
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Compared with Black students in fourth 
grade in the central cities, Black fourth- 
graders in Houston and New York City had 
higher average scores, while Black fourth- 
graders in the District of Columbia had a 
lower average score. Compared to the 
average score of Hispanic fourth-graders in 
the central city schools, no significant 
difference was detected for the average 
score of Hispanic fourth-graders in five of 
the six districts. The average score for 
Hispanic fourth-graders in Los Angeles was 
lower than that in central cities taken as a 
whole. Average scores for White students at 



grade 4 were higher in Atlanta, the District 
of Columbia, and New York City than for 
White students in the central city schools, 
and no significant differences were detected 
between the average scores of White 
students in Chicago, Houston, and Los 
Angeles and the average score in the central 
city schools. Finally, average scores for 
Asian/Pacific Islander students in Los 
Angeles and New York City were not found 
to be significantly different from the average 
score for their counterparts in central cities. 
Sample si 2 es in the other districts were 
insufficient for reliable significance testing. 
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Toble A.3 Students with disabilities and limited English proficient students assessed with accommodations, by type of 
primary accommodation, grade 8: By urban district, 2002 





Urge-print 

book 


Extended 

time 


Weighted percentage of students sampled 

Small Scribe/ 

group One-on-one computer 


— 1 

Other 


SD^ ond/or LEP^ students 


Notion (Public) 


0.03 


1.89 


2.86 


0.11 


0.02 


0.06 


Central city (Public) ^ 


0.01 


2.38 


2.21 


0.09 


0.01 


0.13 


Athnto 


0.09 


# 


0.63 


# 


# 


# 


Chicago 


# 


1.70 


5.39 


# 


# 


# 


District of Columbia 


# 


6.88 


3.19 


# 


# 


# 


Houston 


# 


# 


# 


0.17 


# 


# 


Los Angeles 


# 


1.38 


1.63 


0.06 


# 


0.06 


SD^ students 


Nation (Public) 


0.03 


1.68 


2.76 


0.11 


0.02 


0.06 


Central dty (Public) ^ 


0.01 


2.03 


2.09 


0.09 


0.01 


0.13 


Atlanta 


# 


# 


0.56 


# 




# 


Chicago 


# 


1.55 


5.19 


# 




# 


District of Columbia 


# 


4.91 


2.92 


# 


# 


# 


Houston 


# 


# 


# 


0.17 


# 


# 


Los Angeles 


# 


1.38 


1.56 


0.06 


# 


0.06 


lEP^ students 


Nation (Public) 


# 


0.37 


0.26 


0.01 


# 


0.01 


Central city (Public) ^ 


# 


0.71 


0.29 


0.02 


# 


0.04 


Atbnta 


0.09 


# 


0.07 


# 


# 


# 


Chicago 




0.19 


0.41 


# 


# 


# 


District of Columbia 


# 


2.19 


0.38 


# 


# 


# 


Houston 


# 


# 


# 




# 


# 


Los Angeles 


# 


0.57 


1.07 


# 


# 


0.06 



# Percentage rauruk to zero. 

^ Students wirit disolutties 
^ Limited English prohdent students. 

^ for comparison, ot eighth grode 61 percent of students in centroi city puldit sdtook and 36 percent in ptdilic schools nationolly were non-White. Also, 48 percent of students in central city public schools 
and 34 percent in public schools notionally were eligible for free/reduced-price lunch. 

NOTE: Some students were identihed as bath SD and L£R Such students would be included In bodt the SD and LfP portions of the toble. 

SOURCE: U.S. Department of Education, Institute of biucation Scienos, National Cent» for Biucation Smtistics, Notkinal Assessment of Educational Progress (NAEP), 2002 Trml Urban District Writing 
Assessment. 
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The NAEP assessment may not ;illow 
some accommodations tliat are permitted 
in certain states. For example, in the 
writing assessment, translations of tlie 
prompts into languages other than 
English were not provided. 

Investigating the Potential Effects 
of Exclusion Rates on Assessment 
Results 

The variation in exclusion rates iimong 
states, jurisdictions, and school districts 
may threaten the comparison of results 
within a given year, because the results for 
different districts, states, or jurisdictions 
are based on different proportions ot the 
populations. 



NCES has funded research investigating 
ways in which excluded students might be 
included in the estimation of scores for 
total populations. NCES has dso commis- 
sioned studies on the impact of assessment 
accommodations on overall scores. Several 
statistical approaches for estimating full 
populations (including estimates for ex- 
cluded students) have been proposed using 
data from NhVEP subjects, but none has yet 
been judged completely adequate. Regarding 
the impact of assessment accommodations 
on overall scores, NCES has conducted 
differential item hinctioning (DIF) studies 
of items assessed with and widiout accom- 
modations.^ In these studies, researchers 
found little evidence that accommodations 
changed the functioning of test questions. 
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^ For information on DIF studies of items assessed with accommodations in the 1996 mathematics and science 
assessments, see Nfaziseo, J. M, Carlson, ]. E., Voelkl, KL E,, and Liitkiis, A. D. (1999). Increasing the ’Participation of 
Special Needs Stitdents in NAEP: A Report on 1996 NAEP Research Activities. Washhigton, DC: U.S. Departuieul of 
Education, Office of Educational Research and Improveanent, National Center for Education Statistics. 
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Appendix B 

District-Level Contextual Variables 



To help place results from the NAEP 2002 Trial Urban 
District Assessment into context, this appendix presents 
selected district-level data from sources other than NAEP. 

These data are taken from the Public Ele?nentary/ Secondary 
School Universe Survey, 2()()()-01 , and Uocal liducation Agency 
Universe Survey, 2000— 01 . The interested reader can access 
most of the data presented in this appendix in the NCES 
report, Characteristics of the 100 Uargest Public Elementary 
and Secondary School Districts in the United States at 
http:/ / nces.ed.gov/ pubs2001 /100_largest/ tab_fig.asp. 
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Table BJ Number of students, high-school completers, teachers, ond schools, from non-NAEP sources: 
By urban district, school yeor 2000'-*01 







Number of 


Number of 


1 




Number of 


1999-2000 


full-time equivalent 


Number of 




students^ 


completers^ 


teachers 


schools 


Nationol 


47,086,931 


2,548,076 


2,841,677 


93,344 


Central city 


13,523,126 


610,467 


808,288 


22,310 


Atlanta 


58,230 


2,056 


3,950 


98 


Chicago 


435,261 


14,875 


23,935 


602 


District of Columbio 


68,925 


2,916 


5,044 


165 


Houston 


208,462 


7,735 


11,197 


289 


Los Angeles 


721,346 


27,439 


35,150 


659 


New York City 


1,066,516 


40,827 


65,242 


1,213 



^ Count of students receiving edutotionQl services from school district moy differ somewhat h^om the counts in tobie 8.4, ^ich reflect the count of students from the schools oggregoted up to the school 
(hstrict. 

^ Includes high school diplomo recipients os well os other high school completers (e.g., certificates of attendance), but does not indude high school equivalencies (ex. GEDs). 

SOURCE: U.S. Department of Education, Notionol Center for Educotmn Stotistics. Common Core of Dato, Locol bfuca^on Agency Universe Survey, 2000-01, version la. 



Table B.2 Poverty rate and federol funding, from non-NAEP sources: By urban district, 
1996-97, 1997-98, 2001-02 





5- to 1 7-year olds, 
1999-2000 




Revenue, 

1999-2000 

(in thousonds) 




Title 1 ollocations, I 
2001-02 

(in thousonds) 




Populotion 


Percent 
in poverty 


Totol 


Federol 


Federol os 
percent 
of totol 


Federol 
revenue 
per student 


Bosic 

gronts 


Concentrotion 

gronts 


Atlonto 


66,131 


34.6 


$695,919 


$56,558 


8.1 


$95.17 


$752 


$20,824 


Chicago 


540,667 


24.8 


3,604,873 


539,567 


15 


124.97 


963 


137,865 


District of Columbia 


82,456 


27.8 


881,423 


184,825 


21 


261.19 


1,509 


21,341 


Houston 


230,514 


26 


1,469,074 


152,679 


10.4 


72.8 


585 


50,084 


Los Angeles 


863,656 


28.3 


5,757,819 


592,508 


10.3 


83.45 


668 


178,088 


New York City 


1,397,739 


28.3 


10,945,650 


1,119,944 


10.2 


104.11 


812 


397,541 



NOTE: Federal revenue per student hosed on foil enraOment coOected by the Bureau of the Census. Detail moy not sum to totals, because of to r 
SOURCE: U.S. Department of Education, National Center for Educotion Stotishcs, Common Core of Data survey; ond U.S. Deportment of Commerce, "Survey of Locol Government Rnonces." 
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Table B*3 Number of public elementary and secondary schools, by type of school, from non-NAEP sources: 
By urbon district, school yeor 2000-01 





Total number 
of schools 


Regular 


National 


93,344 


85,493 


Central dty 


22,310 


20,141 


Atlanta 


98 


95 


Chicago 


602 


578 


CNstrkt of Columlua 


165 


150 


Houston 


289 


264 


Los Angeles 


659 


582 


New York City 


1,213 


1,106 



Type of school ' 



Special 

education 


Vocational 

education 


Alternative 


2,008 


1,025 


4,818 


536 


173 


1,460 


0 


0 


3 


24 


0 


0 


10 


0 


5 


1 


3 


21 


18 


0 


59 


8 


18 


81 



^ Type of school is o mutually exclusive category on the Common Core of Dota. There ore coses in v4iich specini education, vocational education, and alternative progroms reside in other types of schools. 
NOTE; Types of schools are dehned in the foHowing way on the Common Core of Data; Regular school— A public elemenniry/secondary school that does not focus primorily on vocotionol, speciol, or 
altemotive education. NAEP is conducted only in regular schods. Specid education schod— A public elementary/secondary schod that (a) focuses pnmarily on specid educotion, including instruchon for 
any of the follovnng; hard of heoring, deof, speech-impaired, health-impaired, onhopedicaHy impaired, men tolly retorded, seriaudy emotion aRy disturbed, multMtandicopped, visually hondicopped, 
deof and blind; and (b) odapts cumcuhim, materials, or instruction for students served. Vocotiond education school— A public elemenlary/secondory schod thot focuses primarily on vocotiond 
educo^on, and provides educotion ond training in one or more ^mi-skdled or technical operotions. Alternative educotion schod— A pudic efementory/secondory school that (o) addresses die needs of 
students that t^icody connot bo met in o regubr school; (b) provides nontroditlond educotion; (c) serves as on adjunct to o regulor school; and |d) foils outside of the cotegories of regular, specid 
educotion, or vocotiond education. 

SOURCE: Ui. Department of Education, Notionol Center for Educotion Stotistics, Common Core of Dom, Public BementoryAecondory School Universe Survey, 2000-01, version 1 o. 
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Table B.4 Number of students in public elementary ond secondary schools, by type of school, from non-NAEP sources: 
By urban district, school year 2000-01 



Type of school' 





Total number 
of students^ 


Regular 


Special 

education 


Vocational 

education 


Other and 
alternative 


Notional 


47,094,888 


46,228,904 


174,577 


199,669 


491,738 


Central city 


13,522,154 


13,152,151 


73,387 


75,953 


220,663 


Atlanta 


58,230 


56,896 


t 


t 


1,334 


Chicago 


435,261 


431,553 


3,708 


t 


t 


District of Columblo 


68,925 


65,285 


2,772 


t 


868 


Houston 


208,462 


204,042 


19 


906 


3,495 


Los Angeles 


721,346 


704,932 


4,480 


t 


11,934 


New York City 


1,066,945 


1,009,319 


1,688 


25,409 


30,529 



d in membership for this type of school. 

^ Typo of school is o muluoily exclusive cotegory on the Common Core of Doto. There ore coses in whidi special educotion, vocotionol education, and alternative progroms reside in other types of 
schools. 

^ Student distribution by type of school is based on memberdilp in the schools of the school district. Counts may vary from those in toble B.1 . 

NOTE: Types of schools ore defined in the following way on the Common Core of Doto: Regular school —A public elementory/secondory school thot does not focus primoriiy on vocotionol, special, or 
alternative educotion. NAEP is conducted only in regutor schools. Special educotion school— A public elementary/suondory school that (o) focuses primarily an specie! education, mduding instruction for 
any of the following: hard of hearing, deof, speechninpaired, health-impoired, orthopedicolty tmfmired, mentally retorded, seriously emotionolly disfurbed, multi-hondicopped, visuaily hondicoj^ed, 
deof and blind; ond (b) adopts curriculum, moten'als, or instruction for students served. Vocotioned education school— A public elementory/secondory school that focuses primorify on vomtional 
education, ond provides education ond troining in one or more sernnkilled or technical (^rations. Altemotira education school— A public elementory/secondory school that (o) addresses the needs of 
students thot lypicoRy cannot be met in o regutor school; (b) provides nontroditionai edocation; (c) serves os on odjunct to o regular school; and (d) falls outside of die categories of regular, special 
education, or vocotionol education. 

SOURCE: U.S. DeparOnent of Educotion, National Center for Education Stotistics, Common Cora of Data, Publit QementaryAecondary School Universe Survey, 2000-01, verson 1o, and Local Education 
f^ency Universe Survey, 200D-01, version lo. 
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Table B.5 Percentoge of minority students, and number of public elementary and secondary schools within specified 
ranges of minority student percentages, from non-NAEP sources: By urban district, school year 2000-01 





Percentage 
of minority 
students 
enrolled 


Number of 
schools with 
students ^ 


Number of schools with a minarity student^ percentoge of | 

0*^20 2M0 41*^60 61-80 81-100 

percent percent percent percent percent 


National 


39.0 


89,110 


43,827 


14,023 


9,866 


7,142 


14,252 


Central city 


63.0 


21,513 


3,534 


3,165 


3,370 


3,088 


8,356 


Atlanta 


93.2 


98 


2 


2 


2 


5 


87 


Chicogo 


90.4 


596 


1 


20 


26 


51 


498 


Distnct of Columl»a 


95.5 


165 


0 


5 


3 


5 


152 


Houston 


90.0 


289 


1 


4 


17 


20 


247 


Los Angeles 


90.1 


659 


4 


16 


50 


88 


501 


New York Gty 


84.7 


1,207 


29 


39 


114 


139 


886 



^ Minorily students, in this toble, includes oil roce/ethnicity cotegofies except White, non-Kisponic. 

^ Indudes only schools for which student membership by roce/ethnicity wos reported. 

SOURCE: U.S. Deportment of Educohon, Notionol Center for Educotion Stotistics, Common Core of Dota, Public Elementory/Secondory School Universe Survey, 2CCfMI, versioii lo, 
and Local Education A^ncy Universe Survey, 2000-01, version 1 o. 



Toble B.6 Number of public elementary and secondary schoals, by Instructional level, from non-NAEP sources: 
By urban district, schaol year 2000-01 





Number of schools 




Number of schoals by instructional leveP 


1 




with students^ 


Primary 


Middle 


High 


Other 


Notionol 


90,711 


52,285 


15,584 


17,280 


5,562 


Central city 


21,900 


13,962 


3,318 


3,081 


1,539 


Atlanta 


98 


70 


15 


12 


1 


Chicago 


596 


472 


21 


72 


31 


District of Columbia 


165 


112 


10 


16 


27 


Houston 


289 


197 


43 


35 


14 


Las Angeles 


659 


456 


75 


105 


23 


New York Gty 


1,207 


692 


210 


184 


121 



^ Instructionol level is bosed on the bwest and highest grade in o school: Primary schools begin between prekin dergorten and grade 3 and may go os high os grade 8. Middle schools hove grade q)ons 
ranging from os low os grade 4 to os high os grade 9. High schoals start at grade 1 or higher and must extend tlvough grade 1 2. Other schools indude oil odier grade combinahons, indudmg 
prekindergorten, kindergorten, or 1-12, and ungraded schools. 

^ indudes only schools for which student membership wos reported. 

SOURCE: U.S. Deportment of Ecbcohan, Nohonol Centm for Education Stohstics, Common Core of Doto, Public Semeniary/Serandory School Universe Survey, 2(H)0-01, version lo, 
and local Educohon Agency Universe Survey, ^00-01, version lo. 
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Table B J Median pupil/teacher ratios in public elementary and secondary schools, by instructlonol level, from non- 
NAEP sources: By urban district, school year 200(H) 1 



Median pupil/teacher ratio' 

By school instructional level^ 





Overall 


Primary 


Middle 


High 


Other 


Notional 


15.9 


16.3 


15.7 


15.1 


12.1 


Centro! city 


16.6 


16.7 


16.0 


17.1 


11.2 


Atlanta 


14.2 


13.7 


16.2 


16.2 


13.2 


Chicago 


18.4 


18.9 


17.3 


15.5 


11.9 


District of Columbia 


13.5 


13.7 


13.6 


13.6 


8.7 


Houston 


18.5 


18.7 


18.7 


18.0 


3.3 


Los Angeles 


19.6 


19.2 


23.6 


22.7 


10.4 


New York Gty 


16.1 


15.7 


16.7 


18.7 


6.3 



' Includes only those schools whose student membership wos greoter than zero. 

^ Instructional level is based on the iovmst ond highest grade offered in o school. Primary schools begin l^tween prelcindergorten ond grade 3 and moy go os high os grade B. Middle sdtaals have grade 
spam ronging from os low os grade 4 to os high os grade 9. High schools start at grode 7 or higher and must extend Hirough grade 12. Other schools include all other grade combinaiions, including 
prekindergarten, kindergarten, ar 1-12, and ungraded schools. 

SOURCE: U.S. Department of Educahon, Nadonal Center for Education Statis^cs, Common Care of Date, Pidrtlc Qemenlary/Secandary Sdtaol Universe Survey, 2000-01, version 1 a, 
and Local Education Agency Universe Survey, 2000-01, version la. 
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Tobie B»8 Percentage of staff in public elementary and secondary schools, by type of staff, from non-NAEP sources: 
By urban district, school year 2000-01 



Percent of full-time equivalent staff 





Total 




Instructional 




staff 


Teachers 


support 


National 


5,397,788 


52.6 


11.4 


Central city 


1,513,730 


53.4 


11.2 


Atlonta 


7,552 


52.3 


12.7 


Chicogo 


28,687 1 


83.4 


2.9 


District of Columblo 


10,808 


46.7 


10.8 


Houston 


24,820 


45.1 


10.2 


Los Angeles 


66,598 


52.8 


14.5 


New York City 


100,198 


65.1 


0.8 



Guidance 

counselors 


Library 
media staff 


District 

administrat05 


School 

administrators 


Other 

staff 


1.8 


1.5 


1.0 


2.5 


29.1 


1.7 


1.3 


0.8 


2.5 


29.1 


2.0 


1.3 


0.5 


2.5 


28.8 


2.9 


1.7 


1.7 


3.1 


4.4 


1.9 


1.3 


0.1 


2.5 


36.8 


1.2 


0.9 


0.1 


2.3 


40.2 


1.4 


0.1 


0.6 


1.9 


28.7 


1.9 


0.7 


0,7 


2.9 


27.9 



^ Hie non-teaching staff categories may b underrepresented. 

NOIf: Percentage may not add to 100, due to rounding. 

SOURCE: Ui. Department of Education, National Center for Education Statistics, Common Core of Onto, LkoI Eduntaon ^ency Universe Survey, 2000-01, version lo. 
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Table B.9 Percentage of reported students eligible for free/reduced’price school lunch and percentage of students in 
each racial/ethnic category, from non^NAEP sources; By urban district, school year 2000-01 



Students eligible for Racial/ ethnic composition of district as 

free/reduced^price lunch ^ a percentage of students enrolled 





Number of 
schools with 
students^ 


Percentage 
of students 


Percentage 
of schools 
reporting 


White 


Black 


Hispanic 


Asian/ 

Pacific 

Islander 


American 

Indian/ 

Alaska 

Native 


Nafionoi 


90,711 


38.6 


83.8 


61.0 


17.0 


16.6 


4.2 


1.2 


Central city 


21,900 


53.6 


82.9 


36.8 


29.2 


27.7 


5.5 


0.9 


Atlanta 


98 


76.4 


100.0 


6.8 


89.5 


2.8 


0.9 


0.1 


Chicago 


596 




0.0 


9.6 


52.0 


34.9 


3.3 


0.2 


District of Columbia 


165 


70.0 


98.8 


4.5 


84.6 


9.2 


1.6 


0.1 


Houston 


289 


70.7 


99.7 


10.0 


32.1 


55.0 


2.9 


0.1 


Los Angeles 


659 


73.5 


99.8 


9.9 


12.8 


70.8 


6.3 


0.3 


New York Gty 


1,207 


71.9 


100.0 


15.3 


34.9 


37.8 


11.7 


0.3 



Not ovoiloble. 

^ These percentages should be interpreted with caution; jurisdictions may not have reported students eligible for reduced-price meals, ond a number of jurisdictions reported participation instead of 
eiigibdity data, which may not be strictly comparable. Percentages are based on tho» schook that reported. 

^ Indudes only schools for which student mendtership wos reported. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Common Core of Data, Public QementoryAecondary Sdiool Universe Survey, 200(M)1, version ] a, 
and Local Education Agency Universe Survey, 2000-01, version In. 
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Appendix C 

Overview of Procedures Used 

for the NAEP 2002 Writing Trial Urban 

District Assessment 



This appendix provides an overview of die NAEP 2002 
writing assessment’s primary components — framework, 
development, adrriinistration, scoring, and analysis. A more 
extensive review of the procedures and methods used in the 
writing assessment can be found in the forthcorning NAEP 
2002 technical report. 

The NAEP 2002 Writing Assessment 

The NAEP 2002 writing assessment is based on the 1998 
writing assessment framework, wEose purpose is to provide, 
based on the expert opinions of writing educators and 
researchers, a definition of writing to guide the NAEP 
writing assessment^ The framework development process 
involved the critical input of hundreds of individuals across 
die country, including representatives of national education 
organizations, teachers, parents, poKcymakers, business 
leaders, and die interested general public. The process was 
managed by the Center for Research on Evaluation, 
Standards, and Student Testing (CRESST) under the 
direction of the National Assessment Governing Board 
(NAGB); the exercise specifications were developed under 
contract by American College Testing (ACT) under die 
direction of NAGB. 



^ Naiioual Assessment Gove tiling Board. (2002). W'niin^ Framework and Spedjt call om for the 
1998 Naiionai Assessment of ’Eihicaiional Frogms. Wasliiiigtou, DC: Audio r. 
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The writing framework delineates six 
major objectives to organize the design of 
the assessment. 

• Students should write for tliree major 
purposes: narrative, informative, and 
persuasive. While other types of writing 
could have been included, the developers 
of the framework believed that, for the 
purpose of monitoring student 
achievement (as opposed to creating 
individual diagnostic assessments), three 
broad types of writing are appropriate. 

• Students should be able to perform a 
variety of writing tasks (letters, essays, 
stories, reports) addressing different 
audiences (peers, school or government 
officials, business representatives). 

• Student writing should be prompted by a 
variety of stimulus materials (letters, 
poems, graphics, reports) under varying 
time constraints. 



• Because writing is a dynamic process 
through which the writer constructs 
meaning, students should develop their 
own writing processes, including methods 
for drafting, evaluating, re^hsing, and 
editing ideas and forms of expression. 
Students are to be given planning space in 
the test materials to generate ideas for 
drafts. In addition, they are gven a 
pamphlet with suggestions for planning, 
revising, and editing. All NAEP student 
responses, given assessment time 
constraints, vire to be evaluated as first 
drafts. 

• Students should display effective choices 
in the organization of their writing. 
Further, they should be able to illustrate 
and elaborate their ideas and should use 
appropriate conventions of English. All 
of these characteristics are to be part of 
die evaluation of student writing. 

• Students should value writing as a 
communicative activity. 

Figure C.l gives examples of various 
writing tasks similar to those included in the 
assessment at grades 4 and 8. Included in 
the figure are descriptions of sample tasks 
that illustrate how each purpose for writing 
is assessed. 
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Figure C\ ifiustrative examples of writing tasks, by purpose for writing, grades 4 and 8 



Purpose for writing 



Grade 4 



Grade 8 



Narrative Provide visual stimuli of o season of the year. 

Ask students to write o letter to o grandparent 
telling the story of on interesting personal 
experience related to the seoson. 



Provide visual stimuli. Ask students to write 
on article for o sports magazine telling the 
story of 0 time when they participated in o 
hobby or skill they enjoyed. 



Informative Provide on appropriate quotation. Ask students 
to explain in on essay to their English teacher 
how 0 person (parent, teacher, friend) has 
influenced them in on important way. 



Provide o series of brief journal entries from 
another historical time. Ask students to explain 
what is revealed about the person who wrote 
the entries. 



Persuasive 



Provide visual stimuli of on animal. Ask 
students to convince their porents/guordions of 
on animal that would moke the best pet. 



Provide brief reviews, os models, of o film, 
TV progrom, or book. Ask students to write 
0 review for the school newspaper that will 
convince other students to watch o favorite 
film or TV program or read o favorite book. 



SOURCE: Hotional Assessment Governor Board. ^Hiug Framework and Spedkations for die / 998 Hational Assessment of Fdacat'mal Progress. Vl^ington, DC: Audwr. 



In addition to the six objectives, the 
framework specifies the percentage ot the 
writing tasks for student responses in the 
assessment that should be devoted to each 
of the three writing purposes — narrative, 
informative, and persuasive. The actiral 
percentage distributions of writing tasks in 
the assessment are listed in table 1.1 of 
chapter 1. The table shows the number of 
tasks at each grade level for each purpose. 
These target percentages vary by grade level 
according to what is deemed 
developmentally appropriate for each grade, 
as st:ated in the writing framework. 



The Assessment Design 

Each student who p^irticipated in the 
writing assessment received a booklet 
containing two 25-minute writing tasks. All 
student responses to the writing tasks were 
rated according to a six-level scoring guide. 
In addition, the test booklets contained 
general background questions and writing- 
specific background questions. 

The assessment design allowed for 
maximum coverage of the wnting domain 
at each grade, while minimriing the time 
burden for any one student This was 
accomplished through the use of matrix 
sampling of tasks, in which each student 
was given only 2 of the 20 or more tasks at 
each grade level. Representative samples of 
students responded to each task, so thatthe 
aggregate results across the entire 
assessment allow broad reporting ot writing 
abilities for the targeted' population. 
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In addition to macrk sampling, the 
assessment design used a procedure for 
distributing blocks across booklets that 
controlled for position and context effects. 
Stxidents received different blocks of tasks 
in tlieir booklets according to a procedure 
called partially balanced incomplete block 
(pBIB) spiraling. The procedure assigned 
blocks of questions in a manner that 
balanced the positioning of blocks across 
booklets and balanced the pairing of blocks 
within booklets according to purposes for 
writing. Blocks were balanced witJiin each 
purpose for writing and were partially 
balanced across purposes for writing. (The 
spiraling aspect of this procedure cycles the 
booklets for administration so that, typically, 
only a few students in any assessment 
session receive die same booklets.) 

In addition to the student assessment 
booklets, three other instruments provided 
data relating to the assessment; — teacher 
questionniiire, a school questionn;iire, and a 
questionn;oire reg;ording students with 
disabilities and limited English proficient 
students (SD/LEP). The SD/LEP 
questionnaire was completed by a school 
staff member knowledgeable about those 
students who were selected to participate in 
the assessment and who were identified as 
having an Individualized Education 
Program (lEP) or equivalent plan and/or 
being limited English proficient (LEP). s\n 
SD/LEP questionnaire was completed for 
each identified student regardless of 
whether the student participated in the 
assessment. Each SD/LEP questionnaire 
asked about the student and the special 
programs in which he or she participated. 



NAEP Samples 

National Sample 

The national results presented in the Nation's 
Report CcV'd: Writing 2002 are based on 
nationally representative probability samples 
of fourth-, eighth-, and twelfth-grade 
students. At grades 4 and 8, the national 
sample was a subset of the combined 
sample of students assessed in each 
participating state ^md the District of 
Columbia, plus an additional sample from 
the states that did not participate in the state 
assessment, and a private school sample. In 
accordance with the NAEP legislation, the 
program uses a random selection process in 
order to obtain a representative sample of 
students for reporting national ;and state or 
jurisdiction results. 

E’ach selected school that participated in 
the assessment and each student assessed 
represents a portion of the population of 
interest. Svimpling weights are needed to 
m^ike valid inferences betw^een the student 
samples and the populations from which 
they were drawn. Sampling weights applied 
to the national and state samples 
compensate for the disproportionate 
representation due to oversampling 
of students who attend nonpublic schools 
and schools with high concentrations of 
Black and/ or Hispanic students. Among 
other uses, sampling weights also 
compensate for lower sampling rates for 
very small schools and are used to adjust for 
school and student nonresponse.^ 
Appropriate sampling weights were 'applied 
to the Trial Urban District Assessment 
S'amples. 



60 



~ Additiouid details regacdiug the design juid structure of the muioual imd state samples may be found in the technicid 
documeutation section of the NAEP web site at hrq)://uces.ed.gov/narionsreporrcai;d. 
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Testing accommodations (e.g., extended 
time, small group testing were permitted 
for special-needs students selected to 
participate in tlie N^VEP writing 
assessments. NAEP inclusion rules were 
applied, and accommodations were oftered 
when a student had an Individualized 
Education Program (lEP) because of a 
disability, was protected under Section 504 
of the Rehabilitation Act of 1973^ because 
of disability, and/or was identified as being 
a limited English proficient student (LEP); 
all other students were asked to participate 
in the assessment under standard 
conditions. 



Standards for Sample 
Participation and Reporting 
of Results 

In carrying out the N^\EP 2002 s tate 
assessments and Trial LTbiin District 
Assessment, the National Center for 
Education Statistics (NCES) established 
participation rate standards that jurisdictions 
were required to meet in order for their 
results to be reported. NCES also 
established additional standards that 
required the annotation of published results 
for jurisdictions whose sample participation 
rates were low enough to raise concerns 
about their representativeness. The NCES 
guidelines used to report results in the state 
assessments and Triiil Urban District 
Assessment, ;and tlie guidelines for notation 
when there is some risk of nonresponse 
bias in the reported results, are presented in 
this section. 



Guideline 1 



The publication of NAEP results I 

The conditions that will result in the publication of a jurisdiction's results ore presented below. 

Guideline 1 — Publication of Public School Results 

A jurisdiction will hove its public school results published in the NAEP 2002 writing report card (or in other reports that include all state* 
level results) if and only if its weighted participation rote for the initial sample of public schools is greater than or equal to 70 percent. 
Similorly, o jurisdiction will receive o separote NAEP stote report if and only if its weighted participation rate for the initial sample of public 
schools is greoter thon or equol to 70 percent. 

Discussion: if o jurisdiction's public school participotion rote for the initiol somple of schools is below 70 percent, there is a substantial 
possibility thot bios will be introduced into the ossessment results. This possibility remoins even ofter moking statistical odjustments to 
compensote for school nonporticipotion. There remoins the likelihood thot, in aggregote, the substitute schools ore sufficiently dissimilar 
from the originols they ore replocing ond represent too greot o proportion of the populotion to discount such o difference. Similarly, the 
ossumptions underlying the use of stotisticol odjustments to compensote for nonporticipotion ore likely to be significontly violated if the 
initiol response rote foils below the 70 percent level. Guideline 1 tokes this into considerotion. This guideline is congruent with current 
NAGB policy, which requires thot doto for jurisdictions thot do not hove o 70 percent before-substitution porticipotion rote be reported "in a 
different formot," ond with the Educotion Information Advisory Committee (EIAC) resolution, which colls for doto from such jurisdictions not 
to be published. 
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^ Sectioa 504 of llie Rfilialiililalioii Act of 1973 is n civil rights law desigued to proliibit discriiiiinatiou on the basis oi 
disabiliU^ in programs and activities, iuclnding education, that receive federal bnancial assistance. 
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The guidelines concerning school and 
student participation rates in the Ni\EP 
state and Trial Urban District Assessments 
were established to address four signific^mt 
ways in which nonresponse bias could be 
introduced into the jurisdiction sample 
estimates: overall school nonresponse, 
strata-specific school nonresponse, overall 
student nonresponse, and strata-specific 



student nonresponse. Presented on the 
following pages are the conditions that will 
result in a jurisdiction or district's receiving a 
notation in the 2002 reports. Note that in 
order for a jurisdiction/district's results to 
be published with no notations, that 
jurisdiction or district must satisfy all 
guidelines. 



Guideline 2 



Reporting school and student participation rates with possible bios due to school nonresponse 
Guideline 2— ‘Notation for Overall Public School Participation Rate 

A jurisdiction that meets Guideline 1 will receive a notation if its weighted participation rate for the initial sample of public schools was 
below 85 percent and the weighted public school participation rate after substitution was below 90 percent. 

Discussion: For jurisdictions that did not use substitute schools, the participation rates are based on participating schools from the original 
sample. In these situations, the NCES standards specify weighted school participation rates of at least 85 percent to guard against potential 
bias due to school nonresponse. Thus the first port of these guidelines, referring to the weighted school participation rate for the initial 
sample of schools, is in direct accordance with NCES standards. 

To help ensure adequate sample representation for each jurisdiction participating in the NAEP 2002 state assessments, NAEP provided 
substitutes for nonparticipating public schools. For jurisdictions that used substitute schools, the assessment results will be based on the 
student data from all schools participating from both the original sample and the list of substitutes (unless both an initial school and its 
substitute eventually participated, in which case only the data from the initial school will be used). 

The NCES standards do not explicitly address the use of substitute schools to replace initially selected schools that decide not to participate 
in the assessment. However, considerable technical consideration was given to this issue. Even though the characteristics of the substitute 
schools were matched as closely as possible to the characteristics of the initially selected schools, substitution does not entirely eliminate 
bias due to the nonparticipation of initially selected schools. Thus, for the weighted school participation rates including substitute schools, 
the guidelines were set ot 90 percent. (NOTE: There was no substitution of schools for the Trial Urban District Assessment) 

If a jurisdiction meets either standard (i.e., 85 percent or higher prior to substitution or 90 percent or higher after substitution), there will 
be no notation for the relevant overall school participation rate. 
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Guideline 3 



Importont segments of the jurisdiction's student population thot 
must be adequately represented to avoid possible nonresponse bias 

Guideline 3— Nototlon for Strata-Spedfic Public School Participation Rates 

A jurisdiction that is not already receiving o notation under Guideline 2 will receive a notation if the sample of public schools included a 
class of schools with similar characteristics that hod o weighted participation rote (after substitution) of below 80 percent, and from which 
the nanparticipating schools together accounted for more than 5 percent of the jurisdiction's total weighted sample of public schools. The 
classes of schools from each of which o jurisdiction needed minimum school participation levels were determined by degree of urbaniza- 
tion, minority enrollment, and median household income of the area in which the school is located. 

Discussion: The NCES standards specify that attention should be given to the representativeness of the sample coverage. Thus, if some 
important segment of the jurisdiction's population is not adequately represented, it is of concern, regardless of the overall participation rate. 

If nonparticipating schools are concentrated within a particular doss of schools, the potential for substantial bias remains, even if the 
overall level of school participation appears to be satisfactory. Nonresponse adjustment cells for public schools have been formed within 
eoch jurisdiction, ond the schools within each cell are similar with respect to degree of urbanization, minority enrollment, and/or median 
household income, os oppropriate for each jurisdiction. 

If the weighted response rate, offer substitution, for a single adjustment cell falls below 80 percent, and more than 5 percent (weighted] of 
the sampled schools ore nonporticipants from such a cell, the potential for nonresponse bias is too great. This guideline is based an the 
NCES standard for stratum-specific school response rates. 

Guideline 4 



Possible student nonresponse bias 

Guideline 4— Nototlon for Overall Student Participation Rote In Public Schools 

A jurisdiction thot meets Guideline I will receive o notation if the weighted student response rate within participating public schools was 
below 85 percent. 

Discussion: This guideline follows the NCES standard of 85 percent for overall student participation rates. The weighted student 
porticipotion rote is bosed on oil eligible students from initlolly selected or substitute schools who participated In the assessment in either an 
initiol session or o moke-up session. If the rote foils below 85 percent, the potential for bias due to students' nonresponse Is too great. 

Guideline 5 



Possible nonresponse bias from inadequately represented strato 

Guideline 5 --Notation for Strata-Specific Student Participation Rates in Public Schools 

A jurisdiction that is not alreody receiving o notation under Guideline 4 will receive o nototion if the sampled students within participating 
public schools included o doss of students with similor choracteristics that had o weighted student response rate of below 80 percent, and 
from which the nonresponding students together occounted for more thon 5 percent of the jurisdiction's weighted assessable public school 
student somple. Student groups from which o jurisdiction needed minimum levels of porticipotion were determined by the oge of the 
student, whether or not the student wos clossified os o student with a disobility (SD) or limited English proficient (LEP), and the type of 
assessment session, os well os school level of urbonizotion, minority enrollment, ond medion household income of the orea in which the 
school is locoted. 

Discussion: This guideline oddresses the foct that if nonparticipating students ore concentroted within o particulor doss of students, the 
potentiol for substontial bios remolns, even if the overoll student porticipotion level oppeors to be sotisfoctory. Student nonresponse 
odjustment cells have been formed using the school-level nonresponse odjustment cells, together with the student's oge ond the noture of 
the ossessment session. 

If the weighted response rote for o single odjustment cell foils below 80 percent, ond more thon 5 percent (weighted] of the invited 
students who do not porticipote in the ossessment ore from such o cell, the potentiol for nonresponse bios is too greot. This guideline is 
bosed on the NCES stondord for strotum-specific student response rotes. 
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In the 2002 Trial Urban District 
Assessment, Nevv York City did not meet 
the initivil public-sdiool participation rate 
standard of 70 percent at the eighth grade. 
Consequently the performance results for 
this grade were not reported. 

At grade 4, New York City also did not 
meet the second guideline (i.e., the weighted 
participation rate for the initial sample of 
schools was below 85 percent and the 
weighted school participation rate after 
substitution was below 90 percent). 

Results for Nevv York City at grade 4 are 
shown with a notation indicating possible 
bias related to nonresponse. 

Trial Urban District Assessment 
Samples 

Sampling for the Trial Urban District 
Assessment was modeled on N^\EP's state 
sampling procedures. However, school 
substitution, which is used in state 
assessments, was not an option in the Trial 
Urbvin District Assessment. Tables C.l and 
C.2 provide a summ:iry of the 2002 urban 
district school and student participation 
rates for the writing assessment sample. 

The first rate in each table is the weighted 



percentage of schools participating in the 
assessment. This rate is based on the 
number of schools that were selected for 
the assessment. The numerator of this rate 
is the estimated number of students 
represented by each selected school that 
participated in the assessment. The 
denominator is the estimated number of 
students represented by each of the selected 
schools that had eligible students enrolled. 

^‘Vlso presented in tables Cl ;ind C.2 are 
weighted student participation rates. The 
numerator of this rate is the estimated 
number of students that each student 
represents across all students assessed (in 
either an initial session or a makeup 
session). The denominator of this rate is the 
estimated number of students represented 
in the sample, across all eligible sampled 
students in p^irticipating schools. The 
number of students that each student 
represents is mainly determined by the 
probability that a student is included in the 
sample, with necessar)^ adjustments made 
for other factors. The ovcnall participation 
rates take into account the weighted 
percentage of student participation after 
makeup sessions. 
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Toble C.1 Weighted school and student participation rates, grade 4: By urban district, 2002 



Grade 4 H 


Number 


Number 


School 


Student 


Overall 




of schools 


of students 


rote 


rote 


rote 


Atlanta 


49 


1,501 


98 


93 


92 


Chicago 


76 


2,037 


95 


92 


87 


Di$tri0 of Columbia 


117 


2,750 


100 


90 


90 


Houston 


49 


1,321 


98 


95 


93 


Los Angeles 


76 


2,005 


100 


93 


93 


New York Gty ^ 


38 


924 


76 


89 


67 



^ Although doemd sufficisnl for reporting, the target res|»)nse rote specified in the NAO’ guidefines was not n^. 

SOURCE: U.S. Department of Education, institute of Education Statistics, Kotionoi Assessm»it of Educational Progress (NAEP), 2002 Trial Urban District Writing Assessment and 2002 Writing Assessement. 



Table C.2 Weighted school ond student participation rotes, grode 8: By urban district, 2002 



Grade 8 ■ 


Number 


Number 


School 


Student 


Overall 




of schools 


of students 


rate 


rate 


rate 


Atlanta 


15 


1,296 


100 


91 


91 


Chicago 


69 


1,611 


94 


92 


86 


District of Columbia 


36 


1,856 


100 


85 


85 


Houston 


34 


1,109 


96 


89 


86 


Los Angeles 


66 


1,778 


97 


90 


87 


New York Gty ^ 


31 


683 


63 


81 


51 



f Imhcotes that the district did not meet the gunlehne for 70 percent school participation in 2002. 

SOURCE: U.S. Department of Education, institute of Educotion Stathrics. National Msessment of Educational Progress (NAEP), 2002 Trial Urbon District Writing Asses^ent end 2002 Writing Assessement. 



Results from the 2002 writing assess- 
ments are reported (on a trial basis) here on 
district-level samples of fourth- and eightli- 
grade students in the kirge urb.in school 
districts that pvirticipated m the Trial Urban 
District Assessment. Results for fourth- and 
eighth-graders in the District of Columbia, 
which has been assessed in the past as a 
jurisdiction, are also reported. The s'^imples 
of students in the urbmi school districts 
represent augmentations of students who 
would normally be selected 21 s part of state 

APPENDIX C • 



samples. These samples allow reliable 
subgroup reporting in these districts. All 
students at “lowed' geographical levels are 
assumed to be part of “higher-level" 
samples. For example, Houston is one of 
the urban districts included in the Triiil 
Urban District Assessment. Data from 
students tested in the Houston sample are 
used to report results for Houston and also 
contribute to the Texas estimates and to tJie 
national cdculations. 
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Tables C.3 and C.4 display die target 
student and school sample sizes planned for 
die Trial Urban District Assessment. The 
first column contains the planned number 
of schools for each district. The second 
column contains the number of schools that 
would have been sampled by NAEP in each 



district had there been no Trial Urban 
District Assessment. The last column shows 
the planned student sample size. Note diat 
the District of Columbia is not presented in 
the tables because its normal sampling plan 
did not have to be adjusted for the Trial 
Urban District Assessment. 



Table C3 Number of schools and students planned for the Trial Urban District Assessment, grade 4: 
By urban district, 2002 





Number of schools 
for trial assessment 


Number of schools 
normally sampled in NAEP 


Number of students 
planned for trial assessment 


Atlanta 


51 


6 


1250 


Chicago 


81 


24 


1900 


Houston 


51 


5 


1250 


Los Angeles 


76 


12 


1900 


New York City ^ 


52 


41 


1250 


t Althougb doomed suffici«it for reporting, the target response rate specified in 


the NAEP guidetmes was not met. 





SOURCE: Ui. Deportment of Eifutotion, Institute of Education Statistics, Notionoi Assessment of bjutationol Progress (NAEP), 2002 Trial Urban District Writing Assessment and 2002 Writing Assessement. 



Table C4 Number of schools and students planned for the Trial Urban District Assessment, grade 8: 
By urbon district, 2002 





Number of schools 
for trial assessment 


Number of schools 
normally sampled in NAEP 


Number of students 
planned for trial assessment 


Atlanta 


17 


8 


1250 


Chicago 


79 


26 


1900 


Houston 


39 


6 


1250 


Los Angeles 


68 


13 


1900 


New York City ^ 


52 


37 


1250 


t Indcotes thot the dhtrict did not meet the guiddina for 70 penent school participation in 2002. 



SOURCE: U.S. Deportment of Education, institute of Education Statistics. National Assessment of Educational Progress (NAEP), 2002 Ind Urlnn District VMting Assessment and 2002 Writing Assessement. 



Note that the sample sizes vary among 
districts. The study was designed to allow 
the examination of the quality of data that 
resulted from samples of different sizes. 



Therefore, larger samples were selected in 
two of the districts (Chicago and Los 
Angeles) iind smaller samples were selected 
in die remaining three. 
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Data Collection and Scoring 

The NAEP 2002 writing assessment was 
conducted from January to March 2002. 
Data collection for the NAEP 2002 
assessment at the national, state, iind district 
levels was conducted by trained field staff 
from Westat. 

Materials from the NAEP 2002 
assessment were shipped to Pearson, where 
trained staff evaluated the responses to the 
constructed-response questions using 
scoring rubrics or guides prepared by 
Educational Testing Service (ETS). All the 
writing tasks were evaluated according to 
six- level scoring guides. At each grade, 
scoring guides were developed for each of 
the three types of writing purpose: 
narrative, informative, and persuasive. 

Specialists in writing who are highly 
experienced in teaching and/or assessing 
wnting trained the professional raters who 
evaluated the student responses. The 
trainers received intensive training together, 
which included reading a manual that 
explained how to use the scoring guides and 
the processes for training and checking 
raters. For each task, or writing problem, the 
trainer, in consultation with other trainers or 
assessment speci^ilists, chose numerous 
sample responses to present to raters and 
prep^ired notes on how the scoring guide 
applied. The sample responses helped raters 
become accustomed to the vimety of 
responses the task elicited before they began 
rating the student responses. Raters had to 
pass a qualifying test before tliey could 



evaluate student responses: they had to 
agree with at least 70 percent of the ratings 
(to a set of ten student responses) that were 
given beforeh'uid by their trainer. In order 
to determine interrater reliability^ of scoring, 
a specified percentage of responses was 
read twice: 6 percent of the responses at 
grades 4 and 8 were read by two raters. 

In 2002, raters scored a total of 608,269 
student responses to writing tasks for the 
nation;il, state, and district assessments. This 
number includes rescoring to monitor 
interrater reliability. The average percentages 
of exact agreement of ratings on the six- 
level scoring guides for the 2002 reliability 
samples were 83 percent at the fourth grade 
and 82 percent at the eighth grade. 

Data Analysis and 

Item Response Theory (IRT) 

Scaling 

After the professional scoring, all 
information was transcribed into the NAEP 
database at ETS. Each processing activity 
was conducted with rigorous quality control. 
After the assessment information was 
compiled in the database, the data were 
weighted according to the population 
structure. The weighting for the national, 
state, and urban district samples reflected 
the probability of selection for each student 
as a result of the sampling design, adjusted 
for nonresponse. 

Analyses were then conducted to 
determine tlie percentages of students 
whose responses to each writing task 



Weigliriug procedures arc described more hilly in tlie “Wsigliring nud Variance Esrimatiou*' section in this document. 
Additional intbrmatiou about the use of weighting procedures may he found in the tedmical docuinentadou section 
of the NiVEP web site at http://nces.ed.gov/ nations reportcard. 
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attained each level on die scoring guide, and 
who provided vanous responses to each 
background question. In calculating 
response percentages for each task, only 
students classified as having been presented 
with the question were included in the 
denominator of the statistic. Students 
whose papers were blank or whose 
responses were judged to be off-topic were 
excluded from the calculation of the scale. 

Item Response Theory (IRT) was used to 
estimate average writing sciile scores for die 
nation, for viirious subgroups ot interest 
within the nation, for die states, and for 
other jurisdictions. IRT models the 
probability of answering a question in a 
certain way as a mathematical function of 
proficiency or skill. NAEP used IRT 
analysis to provide a common scale on 
which performance can be compared 
among groups such as those defined by 
characteristics, including gender and race/ 
ethnicity. 

The results for the NAEP 2002 writing 
assessments are presented on the N^\EP 
writing scales. In 2002, a scale ranging from 
0 to 300 was created to report performance 
at each grade level. The scale summarizes 
student performance across all three 
purposes for writing (narrative, informative, 
and persuasive) in the assessment. 

In producing the writing scale, an IRT 
model was used. The writing tasks (iill rated 
according to six-level scoring guides) were 



scaled by use of a Generalized Partial-Credit 
(GPC) model.^ Developed by ETS and first 
used in 1992, tiie GPC model permits the 
scaling of questions scored according to 
multipoint rating schemes. The model takes 
full advantage of the information available 
from each of the student response 
categories that are used for more complex 
constructed-response questions such as 
writing tasks.^ 

Because of the assessment booklet design 
used by NAEP, students do not receive 
enough wAting tasks to provide reliable 
information about individual performance. 
Traditional test scores for individual 
students, even those based on the IRT, 
would result in misleading estimates of 
population characteristics, such iis subgroup 
means and percentages of students 
performing at or above a certain scale score 
level. However, it is NiVEP’s goal to 
estimate these population ch'.iracteristics. 
NAEP's objectives can be achieved with 
methodologies that produce estimates of 
the population-level parameters directly, 
without the intermediary computation of 
estimates of individuals.' This is 
accomplished using marginal estimation 
sciiling model techniques for latent variables. 
Under the assumptions of the scaling 
models, these population estimates will be 
consistent in the sense that the estimates 
approach the model-based population 
values as the sample size increases. This 
would not be the case for population 



^ Muraki, E. (1992). A Generalize cl Parlial Credit Model; Application of an EM Algorillun. Applied Piychoh^ca! 
Measurement^ i6(2), 159—176. 

^ More detailed information regarding the IRT analyses used in N^VEP may be found in the tedmical documenlation 
section of the NAEP web site at http:// 11 ces.ed.g 0 v/uationst:eportcard. 

' Mislevy, R J., Juid Sheehan, K. M. (1987). Margiiud Estimation Procedures. In A. E. Beaton (Ed.) ImplemefUing the 
New Design: The N-'-iEP 1 983-^1 9S4 Technical (Rep. No. 15-TR-20), pp. 293—260. Princeton, NJ: Educational 
Testing Service. 
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estimates obtained by aggregating optimal 
estimates of individual performance.^ 

Weighting and Variance 
Estimation 

A complex sampling design was used to 
select the students who were assessed. 

The properties of a sample selected through 
such a design may be very different from 
tliose of a simple random sample, in which 
every student in the target population has an 
equal chance of selection and in which the 
observiirions from different sampled 
students can be considered to be statistically 
independent of one another. Therefore, the 
properties of the sample for the data 
collection design were taken into account 
during the analysis of the assessment data. 

One way that the properties of the 
sample design were addressed was by using 
sampling weights to account for the fact 
that the probabilities of selection were not 
identical for .ill students. Al\ population and 
subpopulation characteristics based on the 
assessment data were estimated using 
sampling weights. These weights included 
adjustments for school and student 
nonresponse. 

Not only must appropriate estimates of 
population characteristics be derived, but 
appropriate measures of the degree of 
uncertainty must be obtained for those 
statistics. Two components of uncertainty 



are accounted for in the variability of 
statistics based on student ability: 1) the 
uncertainty due to sampling only a relatively 
small number of students, and 2) the 
uncertainty due to sampling only a portion 
of the cognitive domain of interest. The 
first component accounts for the variability 
associated with the estimated percentages 
of students who had certain background 
characteristics or who had a certain rating 
for their responses to a task. 

Because NAEP uses complex sampling 
procedures, conventional formulas for 
estimating sampling variability daat assume 
simple random sampling are inappropriate. 
Nk\EP uses a jackkiaife replication 
procedure to estimate standard errors. The 
jackknife standard error provides a 
reasonable measure of uncertainty tor any 
student information that can be observed 
without error. However, because each 
student typically responds to only two 
writing tasks, the scale score for any single 
student would be imprecise. In this case, 
N^AEP s marginal estimation methodology 
can be used to describe the performaiace of 
groups and subgroups of students. The 
estimate of tlae variance of the students' 
scale score distributions (which reflect the 
imprecision due to lack of measurement 
accuracy) is computed. This component of 
variability is then included in the standard 
errors of NAEP scale scores.^ 




^ For dieocetical aacl empirical jiistificariou of the procecliices employed, see Mislevy, R. J. (1988). R.-mdomizatioii- 
Based Inferences About Latent Variables from Complex Samples. S6(2), 177—196; Douoghue, J. R. 

(1994). An Empirical Examination of the IRT Information of Polytomously Scored Writing Items Under the 
Generalized Partial Credit Model, journal of Hducatiomd Measurement 51 {f), 295-311. 

^ For furdier detjuls, see Jolmson, E. G., and Rtist, 1C F. (1992). Population Inferences and Variance Estimation for 
NAEP Journal of Educational Statistics /7(2), 175—190. 
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Typically, when the standard error is 
based on a small number of students or 
when the group of students is enrolled in a 
small number of schools, the amount of 
uncertainty associated with the estimation 
of stiindiird errors may be quite large. 
Additional derails concerning procedures 
for identifying such standard errors will be 
found in the technical documentation 
section of the NiVEP web site at http:// 
nces.ed.gov/ nationsreportcard. 

The reader is reminded that, as with 
findings from all surveys, N/VEP results are 
subject to other kinds of error, including 
the effects of imperfect adjustment tor 
student and school nonresponse and 
unloiowable effects associated with the 
particular instrumentation and data 
collection methods. Nonsampling errors can 
be attributed to a number of sources — 
inability to obtain complete information 
about all selected schools in the sample 
(some students or schools refused to 
participate, or students participated but 
answered only certain questions); 
ambiguous definitions; differences in 
interpreting questions; inability or 
unwillingness to give correct background 
information; mistakes in recording, coding, 
or scoring data; and other errors in 
collecting, processing, sampling, and 
estimating missing data. The extent of 
nonsampling errors is difficult to estimate 
and, because of their nature, the impact of 
such errors cannot be reflected in the data- 
based estimates of uncertainty provided in 
N/VEP reports. 



Drawing Inferences 
from the Results 

The reported statistics are estimates and 
are therefore subject to a measure of 
uncertainty. There are two sources of such 
uncertainty. First, N-:\EP uses a sample of 
students rather than testing all students. 
Second, all assessments have some amount 
of uncertainty related to the fact that they 
cannot ask all questions that might be asked 
in a content area. The magnitude of this 
uncertainty is reflected in the estimated 
standard error of each of the estimates. 
When the percentages or average scale 
scores of certain groups are compared, die 
estimated standard error should be taken 
into account, and observed similarities or 
differences should not be relied on solely. 
Therefore, the comp;U‘isons are biased on 
statistical tests that consider the estimated 
standard errors of those statistics and the 
magnitude of the difference among the 
averages or percentages. 

For the data presented in this report, all 
the estimates have corresponding estimated 
standard errors. For example, table C.5 
shows the average scale score for the NAEP 
2002 Trial Urban District Assessments and 
percentages of students at or above 
proficiency levels by gender for grade 4. 
Table C.6 shows the scores anti standard 
errors for the 25th, 50th, 2 ind 75th 
percentiles at grades 4 and 8. Estimated 
standard errors appear in parentheses next 
to each estimated scale score or percentage. 
For the estimated standard errors 
corresponding to other data in this report, 
the reader can consult the NCES web site at 
http:/ / nces.ed.gov/ nationsreportcard/ 
naepdata/. 
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Table C5 Averoge writing scole scores, percentoge of students ot or obove eoch ochievement level, and stondord 
errors, by gender, grade 4 public schools: By urban district, 2002 



il 


Percentage 
1 of sfiidents 


Average 
scole score 


Below Bask 


At or above 
Bask 


At or above 
Profitieaf 


Advanced 


Mole 

Notion (Public) 


51 (0.3) 


144(0.6) 


20(0.5) 


80(0.5) 


18(0.4) 


1(0.1) 


Central city (Public) ^ 


51 (0.4) 


139(0.7)** 


24(0.9) ** 


76(0.9) *" 


15(0.7) *" 


1(0.1) 


Atlanta 


47 0.1) 


131 (2.2) *'** 


31 (3.1) *'*" 


69(3.1) *'** 


8(1.6) *'** 


1 (0.4) 


Chicago 


50 0.1) 


130(1.9) *'** 


32(2.6) *'** 


68(2.6) *'** 


7(1.2) *'** 


#(”*) 


District of Columbia 


490.1) 


127(1.1)*'** 


37(1.5) *'** 


63(1.5) *'** 


7(0.9) *'** 


#(0.2) * 


Houston 


48 0.7) 


141 (2.8) 


24(3.2) 


76(3.2) 


17(2.6) 


1 (0.5) 


Los Angeles 


51 (1.1) 


134(2.1)*'** 


29(2.6) ** 


71 (Z6) *" 


11(1.7) *'** 


#(***) 


New York Qty ^ 


50(2.0) 


145(4.1) 


21 (3.2) 


79(3.2) 


21 (4.1) 


1 (0.6) 


Femole 


Nation (Public) 


49(0.3) 


162(0.4) 


10(0.4) 


90 (0.4) 


35(0.6) 


3(0.2) 


Central city (Public) ^ 


49(0.4) 


155(0.8) ** 


13(0.6) ** 


87(0.6) ** 


27(0.9) ** 


2(0.3) ** 


Atlanta 


530.1) 


147(2.7) *'** 


17(2.9) ** 


83(2.9) ** 


18(2.8) *'** 


2(0.8) 


Chicago 


50 0.1) 


145(2.0) *'** 


17(2.0) ** 


83(2.0) ** 


16(2.6)*'** 


#(***) 


District of Columbia 


510.1) 


143(1.4) *'** 


18(1.5) *'** 


82(1.5) *'** 


15(1.5) *'** 


1 (0.3) *'** 


Houston 


52 0.7) 


154(3.3) ** 


15(2.4) ** 


85(2.4) ** 


28(3.2) 


3(0.9) 


bs Angeles 


490.1) 


148(2.5) *'** 


16(2.3) ** 


84(2.3) ** 


21 (2.6) *'*’ 


1 (0.4) *'*’ 


New York Qty ^ 


50(2.0) 


160(2.9) 


10(3.0) 


90(3.0) 


33(4.0) 


3(1.1) 



# Percentage rounds to zero. 

^ Although deemed suffidoit for reporting, the torget response rote specified in the NAEP guidetines wos not met. 

* Significmitiy dif^rent from central city public schools. 

** ^gmficantlY different from nation (public schools). 

(* ** ) Standmd error estimates can not be accurately determined. 

^ For comparison, at fourth grode 66 percent of students in central city public schools and 40 p^cent in public schools notionally were non-White. Also, 6) percent of students in central city pubhc 
schools and 43 percent in public schools nationoDy were eligible for free/rerhited-price lunch. 

NOIL' Standard errors of the estimated percentoges and scale scores appear in porentheses. Percentages below ond at or obave Basit may not odd to lOO, due to rounding. 

SOURCE: U5. Department of Education, institute of Education Sciences, Notional Center for Education StaHstics, Notional Assessment of ^cottonal Progress (NAEP), 2002 Triol Urban District Writing 
Assessm»it and 2002 VAiting Assessement. 
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Table C.6 Selected writing percentiles and estimated standard errors, grade 4 and 8 public schools: 
By urbon district, 2002 







Scale score distribution 


— ^ — 1 




25fh percentile 


50th percentile 


75fh percentile 


Grade 4 








Nation (Pubhc) 


128(0.6) 


153(0.5) 


178(0.4) 


Central dty (Public) 


1 122(0.7)"* 


146(0.7) "* 


171(0.9) "* 


Atlanta 


117(1.9) *'*" 


139(2.2) "-*" 


161(2.1) 


Chicago 


116(2.6) 


137(1.4) 


160(2.1) 


District of Columbia 


113(1.1)*'*" 


134(1.8) "'*" 


157(1.6) 


Houston 


123(3.7) 


147 (3.0) 


174(4.4) 


Los Angeles 


117(3.0) "* 


141 (1.8) *'*" 


165(2.8) 


New York City 


♦ 128(3.8) 


154(2.51" 


178(4.2) 


Grade 8 








Nation (Public) 


127 (0.7) 


153(0.6) 


178(0.7) 


Central dty (Public) 


2 117(0.8) "" 


144(1.1) "" 


170(1.4) "" 


Atlanta 


107(2.0) 


129(2.2) 


151 (1.6) 


Chicago 


111(3.4) "" 


136(2.2) 


160(3.9) 


District of Columbia 


105(1.7) 


128(1.2) 


152(2.0) 


Houston 


113(4.3) 


139(2.3) "" 


165(3.1) "* 


Los Angeles 


104(1.7) 


128(1.5) 


152(2.0) 



^ Ailiiough deemed sufficient for reporting, the torget response rote specified in die NAEP guidetlnes urns not met. 

* Significontfy different from central city public schoob. 

** Significantly different from notion (public sdioob) 

I For comparison, ot fourth grade 66 percent of students in central dty public schools and ^0 p^cent in public schods nadonolly me ntm-White. Abo, 61 pment of students in control city public schoob 
ond 43 percent In pdilic schools nodonoily were eligible for free/rduced-price lundi. 

^ For comparbon, ot eighth grade 6) percent of students in central city public schools and 36 percent in public schools nationally were non*White. Abo, 48 percent of students in central city public schoob 
ond 34 percent in public schools notionolly were eligible for free/reduceihprice lunch. 

SOURCE: U.S. Deportment of Education, Institute of Education Sciences, National Center for Elucohon Statistics, National Assessment of Educohonol Progress (NAEP), 2002 Triol Urbon District Writing 
Assessment and 2002 Wnting Assessement. 



Using conficience intervals based on the 
standard errors provides a way to take into 
account the uncertainty associated with 
sample estimates ;md to make inferences 
about the population averages and 
percentages in a manner that reflects that 
uncertainty. .An estimated sample average 



scde score plus or minus 1.96 standard 
errors approximates a 95 percent confidence 
inten^al for the corresponding population 
quantity. This statement means that one can 
conclude witli an approximately 95 percent 
level of confidence that the average 
performance of the entire population of 
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interest (e.g., all fourth -grade students in 
public schools) is within plus or minus 1.96 
standard errors of the sample average. 

For example, suppose that the average 
writing scale score of the students in a 
particular group was 156 with an estimated 
standard error of 1.2. ^\n approximately 95 
percent confidence interval for the 
population quantity would be as follows: 

Average [plus or minus] 1.96 standard errors 
156 ± 1.96 X 1.2 
156 ± 2.4 
(153.6, 158.4) 

Thus, one can conclude with a 95 percent 
level of confidence that the average scale 
score for the entire population of students 
in that group is betweenl53.6 and 158.4. It 
should be noted that this example and the 
examples in the following sections are 
illustrative. More precise estimates carried 
out to one or more decimal places are used 
in the actual andyses. 

Similar confidence intervds can be 
constructed for percentages, if the 
percentages are not extremely large or 
extremely small. Extreme percentages 
should be interpreted with caution. Adding 
or subtracting the standard errors associated 
with extreme percentages could cause the 
confidence interval to exceed 100 percent or 
fall below 0 percent, resulting in numbers 
that are not meaningful. A more complete 
discussion of extreme percentages may be 
found in the technical documentation 
section of the Ni\EP web site at http:/ / 
nces.ed.gov/nationsreportcard. 
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Analyzing Group Differences 
in Averages and Percentages 

Statistic;!! tests determine whether the 
evidence based on the d;ita from the groups 
in the sample is strong enough to conclude 
that the averages or percentages are acmally 
different for those groups in the population. 
If the evidence is strong (i.e., the difference 
is statistically significant), the report 
describes tlie group averages or percentages 
as being different (e.g., one group 
performed higher or lower than another 
group), regardless of whether the sample 
averages or percentages appear to be 
approximately the same. The reader is 
cautioned to rely on the results of the 
statistical tests rather than on the app;irent 
magnitude of the difference between 
sample averages or percentages when 
determining whether the sample differences 
are likely to represent ;ictual differences 
;imong the groups in the population. 

To determine whether a real difference 
exists between the average scale scores (or 
percentages of a certain attribute) for two 
groups in the population, one needs to 
obtain an estimate of the degree of 
uncertainty associated with the difference 
between the averages (or percentages) ot 
these groups for the sample. This estimate 
of the degree of uncertainty, called the 
''standard error of the difference” between 
the groups, is obtained by taking the squ;ire 
of each group’s standard error, summing 
the squared standard errors, and taking the 
square root of that sum. 
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Standard Error of tlie Difference — 

se,.3 = TcseT+seT) 

The standard error of the difference, like 
the standard error for an individual group 
average or percentage, can be used to help 
determine whetlier differences among 
groups in the population are real. The 
difference between the averages or 
percentages of the two groups plus or 
minus 1.96 standard errors of the difference 
represents an approximately 95 percent 
confidence inter v;iJ. If the resulting interval 
includes zero, there is insufficient evidence 
to claim a real difference between the 
groups in the population. If the interval 
does not contain zero, the difference 
between the groups is statistic;iJly signitic;int 
at the 0.05 level. 

The following exaniple of comparing 
groups addresses the problem of 
determining whether the average writing 
scale score of group A is higher than that 
of group B. The sample estimates of the 
average scale scores tUid estimated standard 
errors are as follows: 



Group 


Average 


Standard 




Scale Score 


Error 


A 


137 


0.9 


B 


135 


1.1 



The difference betu^een the estimates of 
the average scale scores ot groups A and B 
is two points (137—135). The estimated 
standard error of this difference is 

7 ( 0 . 9 ' + 1 . 1 -) = 



Thus, an approximately 95 percent 
confidence intetwal for this difference is 
plus or minus 1.96 standard errors of the 
difference. 

2 ± 1.96 X 1.4 
2 ± 2.7 
(-0.7, 4.7) 

The value zero is within tlie confidence 
interval; therefore, there is insufficient 
evidence to claim that group A 
outperformed group B. 

Conducting Multiple Tests 

The procedures in the previous section and 
the certainty ascribed to intervals (e.g, a 95 
percent confidence interval) are based on 
statistical theory that assumes that only one 
confidence interv^il or test of statistic^ 
significance is being performed. However, 
there ;ire times when many different groups 
are being compared (i.e., multiple sets of 
confidence intervals are being iinalyzed). In 
sets of confidence intervals, statistical 
theory indicates that the certainty iissociated 
with the entire set of interv 2 ils is less than 
that attributable to each individual 
comparison from the set. To hold the 
significance level for the set of comp:msons 
at a particular level (e.g., 0.05), adjustments 
(called '^multiple comparison procedures^’) 
must be made to the methods described in 
the previous section. One such procedure, 
the Benjamini-Hochberg False Discovery 
l^ite (FDR) procedure was used to control 
the certainty level. 



Miller, R. G. (1981). Sirmdtanmis Statistical Inference (2nd ed.). New York: Springer- Verlag. 

Beujiimmi, Y, and Hocliberg, Y (1993). Controlling die False Discovery Rate: A Practice «uid Powerful Approach to 
Multiple Testing. Journal of the ^Kcyal Statistical Society, Series B, no. 1, 289-300. 
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Unlike the other multiple comparison 
procedures that control the fiimilywise error 
rate (i.e., the probability of making even one 
false rejection in the set of comparisons), 
the FDR procedure controls the expected 
proportion of falsely rejected hypotheses. 
Furthermore, the FDR procedure used in 
NAEP is considered appropriately less 
conservative than familnvise procedures for 
large families of comparisons.^ Therefore, 
the FDR procedure is more suitable for 
multiple comparisons in N^\EP thiin other 
procedures. A detailed description of the 
FDR procedure may be found in the 
technical documentation section of die 



Nh\EP web site at http://nces.ed.gov/ 
nationsreportciird. 

To illustrate how the FDR procedure is 
used, consider the comparisons of male and 
female average writing scale scores for the 
five groups presented in table C.7. Note 
that the difference in average scale scores 
and the estimated stand:u“d error of the 
difference are calculated in a way 
comparable with that of the example in the 
previous section. The test statistic shown is 
the difference in average scale scores 
divided by the estimated standard error of 
the difference. 



Table C J Example of FDR comparisons of average scale scores for different groups of students 





Mole 




Femole 




Mole ond femole 


1 




Average 
scale score 


Standard 

error 


Average 
scale score 


Standard 

error 


Difference 
In averages 


Standard 

error of difference 


Test 

statistic 


Percent 

confidence^ 


Group 1 


224 


1.3 


226 


1.0 


2.08 


1.62 


1.29 


20 


Group 2 


187 


1.7 


193 


1.7 


6.31 


2.36 


2.68 


1 


Group 3 


191 


2.6 


197 


1.7 


6.63 


3.08 


2.15 


4 


Group 4 


229 


4.4 


232 


4.6 


3.24 


6.35 


0.51 


62 


Group 5 


201 


3.4 


196 


4.7 


-5.51 


5.81 


-0.95 


35 



^ The percent confidence is 2(tF{x)) where F(x) is the cumulative distribution of the t-distiibution with the degrees of freedom adjusted to rolled the complexities of the sample design. 



The difference in average senile scores .md 
the estimated standard error of that 
difference can be used to find an 
approximately 95 percent confidence 
interval, as in the example in the previous 
section, or they c;in be used to identify a 
confidence percentage. In the example in 
the previous section, because an 
approximately 95 percent confidence 
interval was desired, the number 1.96 was 
used to multiply the estimated standard 
error of tlie difference to create the 



approximate confidence interval. In the 
current example, the confidence interval for 
the test statistics is identified from statistical 
tables. Instead of checking to see if zero is 
within the 95 percent confidence inter'val 
about die mean, the significance level from 
the statistical tables can be directly 
compared to 100 — 95 — 5 percent. 

If the comparison of average sciile scores 
for mdes and teni:iies were made tor :my 
one of die five groups, there would be a 
significiint difference between the average 




Williams, V. S. L, Jones, L. V., and Tiikcy, J. W (1999). ControUiug Etror in Multiple Comparisons witli Examples 
Fiom State-to-Slate Differences in Educational Acliieveuient . of Bihcational and i^haiioral S taiis tits 24(f), . 
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scale scores for the tvvo gender groups at 
the significance level of less than 5 percent. 
However, because we :ire interested in the 
difference in average scale scores across 
genders for all five of the groups, 
comparing each of the significance levels to 
5 percent is not adequate. Groups of 
students defined by shared characteristics, 
such as racial/ethnic groups, vire treated as 
sets or families when making comparisons. 
However, comparisons of average scale 
scores for each male/ female pair were 
treated separately, so the steps described in 
this example would be replicated for the 
comparison of other male and female 
average scale scores. 

Using the ITD.R procedure to take into 
account tlrat all comparisons are of interest 
to us, the percentages of confidence in the 
example are ordered from Ivirgest to 
smallest: 62, 35, 20, 4, and i. In tlie FDR 
procedure, 62 percent confidence for the 
group 4 comparison would be compared to 
5 percent, 35 percent for the group 5 
compiirison would be compared to 0.05 X 
(5— l)/5 = 0.04 = 4 percent,^^ 20 percent 
for the group 1 comparison would be 
compared to 0.05 X (5’^2)/5 = 0.03 = 3 
percent, 4 percent for the group 3 
comp:irison would be compared to 0.05 X 
(5— 3)/ 5 = 0.02 = 2 percent, and 1 percent 
for the group 2 comparison (actxially slightly 
smaller than 1 prior to rounding) would be 
compared to 0.05 X (5 — 4)/ 5 = 0.01 = 1 
percent. The procedure stops with the first 
contrast found to be significant. If there are 
any remaining contrasts, they are all 
declared to be significant. The last of these 
compiirisons is the only one for which the 



percent confidence is smaller than the FDR 
procedure value. The difference between 
the male :md female average scale scores for 
the group 2 students is significant; for all of 
the other groups, average scale scores for 
males '.md femiiles are not significantly 
different from one another. In practice, a 
very small number of counterintuitive 
results occur when tlie FDR procedures iire 
used to examine differences in subgroup 
results by jurisdiction. In those cases, results 
were not included in this report. NCES is 
continuing to evaluate the use of FDR and 
multiple -comparison procedures for future 
reporting. 

NAEP Reporting Groups 

Results are provided for groups of 
students defined by shared characteristics — 
gender, race/ ethnicity, school’s type of 
location, and eligibility for free /reduced- 
price school lunch. Based on participation 
rate criteria, results are reported for 
subpopulations only when sufficient 
numbers of students and adequate school 
representation are present. The minimum 
requirement is at least 62 students in a 
particular subgroup from at least five 
primary sampling units (PSUs).^*^ The first- 
stage sampling units in the selection of Tnal 
Urban District Assessment samples are 
schools. However, the data for all students, 
regardless of whetJier their subgroup was 
reported separately, were included in 
computing overall results. Definitions of the 
subpopulations are presented below. Note 
that not all of the reporting groups used for 
the national report card are included in this 
report on the urban districts. 



The level ot coufideuce times the number of compurisous minus one divided by die number ot comparisous is 
0.05 (5-l)/5 = 0.04 = 4 percent. 

For the NAEP national assessments prior to 2002, a PSU is a selected geographic region (a comity, group of 
comities, or metropolitan statistical area). In 2002, die first-stage sampling units are scliools (jnihlic suid noupnbiic) 
in the selection of the combined sample. Furdier details about the procedure for determining iiiiuimum sample size 
will appear in teclinical documentadon section of the N/V^Pg^eb site at http://uces.ed.gov/iiatiousreportcard. 
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Gender 

Results are reported separately for males 
and females. 

Race/Efhnicity 

In all NAEP assessments, data about 
student race/ethnicity is collected from two 
sources: school records and student self- 
reports. Previously, NAEP has used student 
self-reported race as the primary race/ 
ethnicity reporting variable. As of 2002, 
school-recorded race has become the race/ 
ethnicity variable presented in NiVEP 
reports. The mutually exclusive racial/ ethnic 
categories are \Sdiite, Black, Hispanic, 
Asian/Pacific Islander, American Indian 
(including .:\laska Native), and Other. When 
a school reports a stxidenfs race as ''Other,” 
tliat category is used. If the school record 
for race is missing for the student, the 
student’s response to the race/ethnicity 
question is then used. If student data are 
missing (i.e. the student did not respond or 
gvive multiple responses), then the student is 
coded to the "Other/ missing” category. The 
combination of these 2 sets of student 
categories is used for the "other” category. 
The race /ethnicity tables in this report omit 
the "Other” category because the 
percentages were found to be consistently 
under one percent. Information based on 
student self-reported race/ethnicity will 
continue to be available on the N/VEP Data 
Tool (http://nces.ed.gov/ 
nationsreportcard/ naepdata/). 

Type of Location 

In most NAEP assessments, results are 
reported for students attending schools in 
tliree mutually exclusive location types: 
central city, urban fringe /large town, and 
airal/small town. Results for tlae NAEP 



2002 Trial Urban District Assessment are 
reported for students attending schools in 
one type of location — central city. Follow- 
ing standard definitions established by the 
Federal Office of Management and Budget, 
the U. S. Census Bureau (see http:/ / 
www.census.gov/) defines "central city” as 
the largest city of a Metropolitan Statistical 
^\rea (MSA) or a Consolidated Metropolitan 
Statisticiil ^\rea (CMSA). An MSA is an area 
defined by the federal government for the 
purposes of presenting general-purpose 
statistics for metropolitan areas. Typically, an 
MSA contains a city with a population of at 
least 50,000 and includes its adjacent areas. 
An MSA becomes a CMSA if it meets the 
requirements to qualify as a metropolitan 
statistical area, has a population of 
1,000,000 or more, its component parts are 
recognized as prim:u'y metropolitan statisti- 
cal areas, and local opinion tavors the 
designation. 

In the NCES Common Core of Data 
(CCD) locale codes are assigned to schools. 
For the definition of centriil city used in this 
report, two locale codes of the survey are 
combined. The definition of each school’s 
type of location is determined by the size 
of the place where the school is located and 
whether or not it is in an MSA or CMSA. 
School locale codes are assigned by the US. 
Bureau of the Census. For the definition of 
central city N^\EP reporting uses data from 
two CCD locale codes: large city (a central 
city'' of an MSA or CMSA with the city 
having a population greater than or equal to 
250,000) and midsize city (a central city of 
a MSA or CMSA having a population less 
than 250,000). Central city is a geographical 
term and is not synonymous with "inner city.” 
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The boundaries of an urban school 
district and a city may not always coincide. 
Los Angeles Unified, for example, extends 
beyond the city boundaries and includes 
urban fringe areas of the MSA, although the 
entire district is coded as central city. Most 
of the other districts included in this report 
have school districts that share the same 
boundaries as the city. Tlie interested reader 
may view the School District Demographics 
website at http:/ / nces.ed.gov/surveys/ sdds, 
where the school district's boundaries can 
be shown on a map that also has county 
boundaries. In the Houston and Los Ange- 
les districts, some students attended schools 
located in the category ot urban fringe/ 
large town. These included 5 percent and 19 
percent of fourth-grade students in Hous- 
ton and Los ^Migeles respectively, as well as 
24 percent of grade 8 students in Los 
Angeles. Urb;in fringe /large town is a 
NAEP classification that combines three 
categories: Urban fringe of large city, urban 
fringe of midsize city, and large town. ^\n 
urban fringe includes all densely settled 
places within MSAs that arc classified as 
urban by the LIS. Census Bureau. A large 
town is defined as a place outside MSAs 
with a population of less than 25,000 but 
greater than or equal to 2,500. Across the 
tot:al sample for the N^AEP writing assess- 
ment in 2002, 28 percent of students 
attended schools classified as central city, 42 
percent attended schools classified as urban 
fringe /l;arge town, and 30 percent attended 
schools classified as rural. 



Eligibility for Free/ 

Reduced^Price School Lunch 

Based on available school records, 
students were classified either as currently 
eligible for the free/reduced-price school 
lunch component of the US. Department 
of Agriculture's National School Lunch 
Program or as not eligible. Eligibility for the 
program is determined by a student's family 
income in relation to the federally 
established poverty level. Free lunch 
qu'Jification is set at 130 percent of the 
poverty level, and reduced-price lunch 
qualification is set at 170 percent of the 
poverty level. The classification applies only 
to the school year when the assessment was 
administered (i.e., the 2001-02 school year) 
and is not based on eligibility in previous 
years. If school records were not available, 
the student was classified as '"Information 
not available." If the school did not 
p^irticipate in the program, all students in 
that school were classified as ""Information 
not av;ailable." 
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